-----============= acceptance-small: sanity-flr ============----- Mon Aug 10 18:13:14 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg337-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 18:13:32 (1786400012) === oleg337-client.virtnet: executing check_config_client /mnt/lustre oleg337-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg337-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8ad60a2f5000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8ad60a2f5000.idle_timeout=debug disable quota as required oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 18:14:02 (1786400042) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7620] == sanity-flr test 0a: lfs mirror create with -N option == 18:14:04 (1786400044) PASS 0a (14s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 18:14:18 (1786400058) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 18:14:21 (1786400061) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 18:14:25 (1786400065) PASS 0d (19s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 18:14:44 (1786400084) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 18:14:48 (1786400088) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 18:14:51 (1786400091) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8ad60a2f5000.stats=clear osc.lustre-OST0001-osc-ffff8ad60a2f5000.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 18:14:58 (1786400098) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 18:15:08 (1786400108) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 18:15:16 (1786400116) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 18:15:20 (1786400120) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 18:15:29 (1786400129) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 18:15:38 (1786400138) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 18:15:45 (1786400145) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 18:15:47 (1786400147) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 18:15:55 (1786400155) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 18:16:04 (1786400164) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 18:16:11 (1786400171) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 1.01699 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829914 s, 12.6 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 18:16:21 (1786400181) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 0.248356 s, 76.0 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 18:16:28 (1786400188) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 18:16:37 (1786400197) Stopping /mnt/lustre-ost1 (opts:) on oleg337-server oleg337-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ad60a2f5000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ad60a2f5000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg337-server: oleg337-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg337-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg337-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg337-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg337-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg337-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg337-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg337-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg337-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg337-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg337-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg337-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg337-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg337-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg337-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg337-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg337-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg337-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg337-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg337-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg337-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg337-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg337-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg337-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg337-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg337-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg337-server: Waiting 1410 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg337-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg337-server oleg337-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8ad60a2f5000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ad60a2f5000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg337-server: oleg337-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg337-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (125s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 18:18:42 (1786400322) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.366518 s, 22.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg337-server oleg337-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ad60a2f5000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ad60a2f5000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 8388608 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg337-server: oleg337-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg337-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (36s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 18:19:18 (1786400358) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg337-server oleg337-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ad60a2f5000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ad60a2f5000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg337-server: oleg337-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg337-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg337-server oleg337-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8ad60a2f5000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ad60a2f5000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg337-server: oleg337-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg337-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (67s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 18:20:26 (1786400426) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.81848 s, 13.7 MB/s -rw-r--r-- 1 root root 52428800 Aug 10 18:20 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.934s user 0m0.000s sys 0m0.178s Stopping /mnt/lustre-ost1 (opts:) on oleg337-server oleg337-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ad60a2f5000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ad60a2f5000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.117s user 0m0.001s sys 0m0.189s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg337-server: oleg337-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg337-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg337-server oleg337-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8ad60a2f5000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ad60a2f5000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.152s user 0m0.001s sys 0m0.150s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg337-server: oleg337-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg337-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg337-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8ad60a2f5000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ad60a2f5000.ost_server_uuid in FULL state after 0 sec PASS 33b (82s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 18:21:48 (1786400508) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 18:21:51 (1786400511) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 18:21:54 (1786400514) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 18:21:57 (1786400517) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 18:22:05 (1786400525) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.48483 s, 26.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 18:22:17 (1786400537) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212441 s, 49.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279667 s, 37.5 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 18:22:25 (1786400545) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 18:22:34 (1786400554) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0010138 s, 1.0 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00124534 s, 822 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00114995 s, 890 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00105357 s, 972 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 18:22:45 (1786400565) create test files with size 15748048 .. fe0048fde0482efac4040fc6144f6f52 - 44c8797144a6d06e90b4008cfa369113 - 242f2f7a71114f8c50b84c6c55c7ed44 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg337-client.virtnet /mnt/lustre (opts:) Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre PASS 37 (27s) == sanity-flr test 38: resync ============================ 18:23:12 (1786400592) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 18:23:26 (1786400606) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 18:23:33 (1786400613) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194005 s, 5.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244457 s, 42.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 18:23:44 (1786400624) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.101482 s, 20.7 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.29656 s, 14.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.161246 s, 13.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (16s) == sanity-flr test 42: lfs mirror verify ================= 18:24:00 (1786400640) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 18:24:04 (1786400644) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 18:24:07 (1786400647) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283334 s, 37.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 18:24:15 (1786400655) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 18:24:19 (1786400659) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe4:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 18:24:29 (1786400669) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.392094 s, 26.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg337-client.virtnet /mnt/lustre (opts:) Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 18:24:40 (1786400680) llite.lustre-ffff8ad60b0c1000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.255365 s, 41.1 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg337-client.virtnet /mnt/lustre (opts:) Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8ad60c083000.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 18:24:53 (1786400693) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (30s) == sanity-flr test 46: Verify setstripe --copy option ==== 18:25:23 (1786400723) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (32s) == sanity-flr test 47: Verify mirror obj alloc =========== 18:25:55 (1786400755) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 18:25:59 (1786400759) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.14534 s, 21.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.177848 s, 17.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xd5af90e6 (nosync,stale) Mirror 2: 0xd5af90e6 (nosync,stale) ** verify mirror 2 doesn't change original checksum: fb204fdd6a04b67f61ce1b75264223ba - mirror 1 checksum: f6063bad630cf49a62d713edb14a022b - mirror 2 checksum: fb204fdd6a04b67f61ce1b75264223ba - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-10T18:26:02-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-10T18:26:02-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: fb204fdd6a04b67f61ce1b75264223ba - mirror 1 checksum: f6063bad630cf49a62d713edb14a022b - mirror 2 checksum: f6063bad630cf49a62d713edb14a022b - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-10T18:26:02-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-10T18:26:02-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } PASS 48 (10s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 18:26:11 (1786400771) Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server 18:26:19 (1786400779) shut down facet: mds1 facet_host: oleg337-server facet_failover_host: oleg337-server Failover mds1 to oleg337-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 18:26:50 (1786400810) targets are mounted 18:26:50 (1786400810) facet_failover done oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255496 s, 41.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe0:0x0] } PASS 50A (57s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 18:27:07 (1786400827) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 18:27:11 (1786400831) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 18:27:15 (1786400835) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00730211 s, 2.2 MB/s PASS 50c (6s) == sanity-flr test 50d: mirror rsync keep holes ========== 18:27:22 (1786400842) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 18:27:25 (1786400845) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (4s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 18:27:29 (1786400849) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (20s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 18:27:49 (1786400869) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 18:28:18 (1786400898) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 18:28:42 (1786400922) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 18:29:07 (1786400947) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 18:29:16 (1786400956) mirror create pid 53771 mirror split pid 53772 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 53771 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 53772 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 18:30:23 (1786401023) Seed set to 1025 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000003 1786401025.603780 write 0x011abe thru 0x017f5d (0x64a0 bytes) truncating to largest ever: 0x24951 000005 1786401025.606487 trunc from 0x017f5e to 0x024950 (0xc9f3 bytes) 000007 1786401025.623661 read 0x01c0ef thru 0x01ff5d (0x3e6f bytes) 000008 1786401025.629329 mapwrite 0x0336c9 thru 0x03afc6 (0x78fe bytes) 000011 1786401025.707493 read 0x00a514 thru 0x0168a1 (0xc38e bytes) 000012 1786401025.722107 read 0x002cdf thru 0x00b8d2 (0x8bf4 bytes) 000015 1786401025.724638 read 0x031798 thru 0x03afc6 (0x982f bytes) 000016 1786401025.725698 trunc from 0x03afc7 to 0x00fe79 (0x2b14d bytes) 000019 1786401025.750830 mapwrite 0x03d82a thru 0x03da78 (0x24f bytes) 000022 1786401025.813710 mapread 0x026ff8 thru 0x02fbfe (0x8c07 bytes) 000024 1786401025.816294 mapwrite 0x022d59 thru 0x02d255 (0xa4fd bytes) truncating to largest ever: 0x3d337 000025 1786401025.865823 trunc from 0x03da79 to 0x03d336 (0x742 bytes) 000026 1786401025.880719 trunc from 0x03d337 to 0x007293 (0x360a3 bytes) 000027 1786401025.896334 read 0x00475a thru 0x007293 (0x2b3a bytes) 000030 1786401025.898849 mapwrite 0x017f25 thru 0x0263ec (0xe4c8 bytes) 000031 1786401025.960707 trunc from 0x0263ed to 0x015b83 (0x10869 bytes) 000032 1786401025.975189 read 0x00f844 thru 0x015b83 (0x6340 bytes) 000033 1786401025.976563 mapread 0x00f91d thru 0x0144b7 (0x4b9b bytes) 000037 1786401025.982655 read 0x00fd5e thru 0x0107fa (0xa9d bytes) 000039 1786401025.984353 mapwrite 0x00e3fc thru 0x0138a7 (0x54ac bytes) 000040 1786401026.053263 MIRROR_EXTEND to 2 mirrors 000041 1786401026.601318 mapwrite 0x03d25c thru 0x03ffff (0x2da4 bytes) 000042 1786401026.732858 read 0x01a000 thru 0x021d71 (0x7d72 bytes) 000044 1786401026.742835 trunc from 0x040000 to 0x012bb9 (0x2d446 bytes) 000045 1786401026.758988 read 0x004b98 thru 0x00722a (0x2693 bytes) 000046 1786401026.768079 mapwrite 0x013fa6 thru 0x01d85c (0x98b7 bytes) 000048 1786401026.826689 write 0x00557a thru 0x00a213 (0x4c9a bytes) 000049 1786401026.831917 mapread 0x009c8c thru 0x0150c5 (0xb43a bytes) 000050 1786401026.836782 write 0x0129e5 thru 0x01b53e (0x8b5a bytes) 000052 1786401026.841170 mapwrite 0x01c402 thru 0x026ba7 (0xa7a6 bytes) 000053 1786401026.917804 read 0x024449 thru 0x026ba7 (0x275f bytes) 000055 1786401026.923147 MIRROR_RESYNC 2 mirrors 000058 1786401027.383459 mapread 0x0042cb thru 0x007e6d (0x3ba3 bytes) 000059 1786401027.409543 read 0x001655 thru 0x002715 (0x10c1 bytes) 000060 1786401027.410033 read 0x01811e thru 0x018600 (0x4e3 bytes) 000061 1786401027.425111 read 0x0014fb thru 0x007a9a (0x65a0 bytes) 000062 1786401027.426000 write 0x01604f thru 0x022c73 (0xcc25 bytes) 000063 1786401027.493973 mapwrite 0x003144 thru 0x006f20 (0x3ddd bytes) 000065 1786401027.540874 mapwrite 0x0305bb thru 0x03db44 (0xd58a bytes) 000066 1786401027.637776 mapread 0x02788e thru 0x02fec4 (0x8637 bytes) 000067 1786401027.641027 write 0x020f3f thru 0x029e79 (0x8f3b bytes) 000068 1786401027.651861 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) 000071 1786401027.827696 MIRROR_EXTEND to 2 mirrors 000075 1786401028.623455 read 0x021008 thru 0x028706 (0x76ff bytes) 000077 1786401028.644821 write 0x02929d thru 0x02e62a (0x538e bytes) 000078 1786401028.733851 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) 000080 1786401028.915887 trunc from 0x03db45 to 0x00e9e4 (0x2f160 bytes) 000081 1786401028.937672 mapread 0x00a047 thru 0x00e9e4 (0x499e bytes) 000082 1786401028.954306 read 0x0041cb thru 0x00673e (0x2574 bytes) 000084 1786401028.959481 write 0x00013e thru 0x00ce3d (0xcd00 bytes) 000089 1786401028.976409 MIRROR_EXTEND to 2 mirrors 000090 1786401029.408191 trunc from 0x00e9e5 to 0x02668e (0x17caa bytes) 000093 1786401029.464115 MIRROR_RESYNC 2 mirrors 000094 1786401029.953248 mapread 0x022b22 thru 0x02668e (0x3b6d bytes) 000096 1786401029.974135 mapread 0x01166f thru 0x011c8f (0x621 bytes) 000099 1786401029.991234 write 0x03b1e5 thru 0x03ffff (0x4e1b bytes) 000100 1786401030.057259 trunc from 0x040000 to 0x006581 (0x39a7e bytes) 000101 1786401030.086698 mapwrite 0x00d577 thru 0x0171cc (0x9c56 bytes) 000102 1786401030.178890 mapwrite 0x03d4a7 thru 0x03ffff (0x2b59 bytes) 000103 1786401030.255787 write 0x0291ae thru 0x03077c (0x75cf bytes) 000104 1786401030.272219 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x293df6e2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe6c57c54 Mirror 3: 0xe6c57c54 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000105 1786401030.919741 mapwrite 0x0017f6 thru 0x005fa8 (0x47b3 bytes) 000107 1786401031.087632 read 0x00bdd6 thru 0x0150c6 (0x92f1 bytes) 000110 1786401031.101473 mapwrite 0x006b7e thru 0x011329 (0xa7ac bytes) 000111 1786401031.197778 mapwrite 0x025780 thru 0x029846 (0x40c7 bytes) 000112 1786401031.282993 mapread 0x0033f9 thru 0x008f33 (0x5b3b bytes) 000115 1786401031.285754 mapread 0x017143 thru 0x026f0c (0xfdca bytes) 000118 1786401031.312323 write 0x003d32 thru 0x00bd54 (0x8023 bytes) 000119 1786401031.315209 read 0x037281 thru 0x03ffff (0x8d7f bytes) 000120 1786401031.326673 mapwrite 0x018921 thru 0x01fe4e (0x752e bytes) 000121 1786401031.385735 trunc from 0x040000 to 0x02ca17 (0x135e8 bytes) 000122 1786401031.431590 write 0x02b2ba thru 0x033200 (0x7f47 bytes) 000124 1786401031.434222 read 0x00596b thru 0x00dfa9 (0x863f bytes) 000125 1786401031.439858 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33201): Mirror 2: 0x8cb7c79e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33201): Mirror 2: 0x8cb7c79e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33201: skipped 000126 1786401031.789168 read 0x00b7a2 thru 0x013b6a (0x83c9 bytes) 000133 1786401031.805490 trunc from 0x033201 to 0x00783e (0x2b9c2 bytes) 000134 1786401031.918414 MIRROR_RESYNC 2 mirrors 000135 1786401032.556421 read 0x0075a7 thru 0x00783e (0x298 bytes) 000136 1786401032.565086 trunc from 0x00783f to 0x027312 (0x1fad4 bytes) 000137 1786401032.704928 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) 000140 1786401032.928668 trunc from 0x027313 to 0x031a64 (0xa752 bytes) 000141 1786401032.967151 mapwrite 0x029dd2 thru 0x02c2c5 (0x24f4 bytes) 000142 1786401033.088995 mapwrite 0x020ac1 thru 0x024414 (0x3954 bytes) 000144 1786401033.160009 read 0x01d6f6 thru 0x02c684 (0xef8f bytes) 000145 1786401033.160832 read 0x0240e5 thru 0x02b7b4 (0x76d0 bytes) 000147 1786401033.162608 read 0x02ca94 thru 0x031a64 (0x4fd1 bytes) 000148 1786401033.163591 write 0x02e52f thru 0x032f52 (0x4a24 bytes) 000152 1786401033.169348 mapwrite 0x00fc6b thru 0x01606f (0x6405 bytes) 000153 1786401033.236531 write 0x01be12 thru 0x027a45 (0xbc34 bytes) 000154 1786401033.238503 mapwrite 0x01fb57 thru 0x020773 (0xc1d bytes) 000157 1786401033.286119 write 0x032061 thru 0x03226c (0x20c bytes) 000158 1786401033.287428 mapwrite 0x03a7bf thru 0x03ffff (0x5841 bytes) 000159 1786401033.443256 mapread 0x034af3 thru 0x03c0ea (0x75f8 bytes) 000161 1786401033.445754 trunc from 0x040000 to 0x027a26 (0x185d9 bytes) 000162 1786401033.467124 mapread 0x017749 thru 0x0179b9 (0x271 bytes) 000165 1786401033.470524 mapwrite 0x02a901 thru 0x03982d (0xef2d bytes) 000167 1786401033.755948 mapwrite 0x000a7e thru 0x00d87c (0xcdff bytes) 000169 1786401033.813584 read 0x02c7ed thru 0x02d339 (0xb4d bytes) 000170 1786401033.814445 mapread 0x0053ab thru 0x009df1 (0x4a47 bytes) 000172 1786401033.816566 mapread 0x016ce2 thru 0x0258d5 (0xebf4 bytes) 000174 1786401033.820227 write 0x0177ab thru 0x026476 (0xeccc bytes) 000175 1786401033.822672 write 0x037b23 thru 0x03ffff (0x84dd bytes) 000177 1786401033.830243 MIRROR_EXTEND to 2 mirrors 000180 1786401034.517412 mapread 0x01bf31 thru 0x024133 (0x8203 bytes) 000181 1786401034.550980 write 0x02c01f thru 0x02e13b (0x211d bytes) 000183 1786401034.638775 mapread 0x02b385 thru 0x038c61 (0xd8dd bytes) 000185 1786401034.670592 mapread 0x03d5ac thru 0x03d9e2 (0x437 bytes) 000186 1786401034.682830 read 0x033558 thru 0x0349ac (0x1455 bytes) 000187 1786401034.692232 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) 000188 1786401034.938090 mapread 0x0297cb thru 0x033937 (0xa16d bytes) 000190 1786401034.977020 trunc from 0x040000 to 0x00f63c (0x309c3 bytes) 000192 1786401035.039899 trunc from 0x00f63d to 0x03994a (0x2a30e bytes) 000193 1786401035.062935 mapread 0x01a8df thru 0x028be0 (0xe302 bytes) 000195 1786401035.087782 trunc from 0x03994b to 0x027645 (0x12305 bytes) 000196 1786401035.105658 mapwrite 0x036749 thru 0x03ffff (0x98b7 bytes) 000198 1786401035.390296 mapwrite 0x00916d thru 0x01604b (0xcedf bytes) 000202 1786401035.514875 write 0x038153 thru 0x03ffff (0x7ead bytes) 000204 1786401035.517193 mapwrite 0x01f534 thru 0x029ab1 (0xa57e bytes) 000208 1786401035.578847 trunc from 0x040000 to 0x029e63 (0x1619c bytes) 000212 1786401035.597067 read 0x011b7f thru 0x01af5b (0x93dd bytes) 000213 1786401035.598136 mapwrite 0x00a4b6 thru 0x01299f (0x84ea bytes) 000214 1786401035.667949 trunc from 0x029e64 to 0x01c2e2 (0xdb81 bytes) 000215 1786401035.688544 mapwrite 0x030299 thru 0x033aea (0x3852 bytes) 000218 1786401035.759840 trunc from 0x033aeb to 0x01fe86 (0x13c64 bytes) 000223 1786401035.807150 read 0x014021 thru 0x019c31 (0x5c11 bytes) 000226 1786401035.812911 trunc from 0x01fe87 to 0x018867 (0x761f bytes) 000227 1786401035.865710 trunc from 0x018868 to 0x007db3 (0x10ab4 bytes) 000228 1786401035.923989 read 0x002b90 thru 0x00363d (0xaae bytes) 000230 1786401035.948883 write 0x035e5b thru 0x03d9b6 (0x7b5c bytes) 000231 1786401035.953983 mapwrite 0x0061a6 thru 0x01496a (0xe7c5 bytes) 000236 1786401036.066453 read 0x03b821 thru 0x03d9b6 (0x2196 bytes) 000237 1786401036.067199 mapread 0x0047b8 thru 0x0120b2 (0xd8fb bytes) 000238 1786401036.068375 mapwrite 0x029014 thru 0x02e77a (0x5767 bytes) 000239 1786401036.103229 mapwrite 0x00c640 thru 0x017e78 (0xb839 bytes) 000240 1786401036.177212 read 0x03bc46 thru 0x03d9b6 (0x1d71 bytes) 000243 1786401036.180775 trunc from 0x03d9b7 to 0x03004e (0xd968 bytes) 000244 1786401036.233604 trunc from 0x03004f to 0x03c909 (0xc8bb bytes) 000245 1786401036.251883 mapread 0x01e234 thru 0x0222df (0x40ac bytes) 000246 1786401036.260920 MIRROR_EXTEND to 2 mirrors 000248 1786401036.703552 mapread 0x02affb thru 0x031f83 (0x6f89 bytes) 000249 1786401036.724541 mapread 0x00ab47 thru 0x013989 (0x8e43 bytes) 000250 1786401036.745297 mapwrite 0x0218c8 thru 0x02418b (0x28c4 bytes) 000251 1786401036.869576 write 0x028870 thru 0x02d8de (0x506f bytes) 000252 1786401036.875188 mapread 0x03822a thru 0x03c909 (0x46e0 bytes) 000253 1786401036.884727 trunc from 0x03c90a to 0x037686 (0x5283 bytes) 000254 1786401036.922786 mapwrite 0x008d88 thru 0x0126a1 (0x991a bytes) 000258 1786401036.979201 read 0x02c18d thru 0x035797 (0x960b bytes) 000259 1786401036.984187 mapwrite 0x008923 thru 0x00b2f4 (0x29d2 bytes) 000260 1786401037.020243 read 0x034d1e thru 0x037686 (0x2969 bytes) 000261 1786401037.021847 write 0x0143d6 thru 0x01596b (0x1596 bytes) 000262 1786401037.023286 write 0x00875c thru 0x008d52 (0x5f7 bytes) 000263 1786401037.024891 read 0x0192ae thru 0x01ea5d (0x57b0 bytes) 000264 1786401037.026276 mapread 0x0163d2 thru 0x022f66 (0xcb95 bytes) 000267 1786401037.030011 read 0x00e2df thru 0x01375f (0x5481 bytes) 000269 1786401037.033106 trunc from 0x037687 to 0x027aa5 (0xfbe1 bytes) 000276 1786401037.084833 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x27aa6): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27aa6): Mirror 2: 0x71dc50b3 (stale) CRC-32 checksum value for chunk [0, 0x27aa6): Mirror 1: 0x1b0791c0 Mirror 3: 0x1b0791c0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27aa6: skipped 000277 1786401037.677947 MIRROR_RESYNC 3 mirrors 000278 1786401038.305185 write 0x006805 thru 0x015824 (0xf020 bytes) 000279 1786401038.449690 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27aa6): Mirror 2: 0x1b0791c0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27aa6): Mirror 2: 0x1b0791c0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27aa6: skipped 000281 1786401038.803032 write 0x02b19c thru 0x02b488 (0x2ed bytes) 000282 1786401038.804566 read 0x0119a8 thru 0x013a2d (0x2086 bytes) 000283 1786401038.809579 trunc from 0x02b489 to 0x039e9f (0xea17 bytes) 000286 1786401038.829145 MIRROR_RESYNC 2 mirrors 000287 1786401039.436763 write 0x034330 thru 0x03e8ac (0xa57d bytes) 000288 1786401039.690512 MIRROR_RESYNC 2 mirrors 000289 1786401040.220272 mapwrite 0x024d97 thru 0x02a535 (0x579f bytes) 000290 1786401040.361911 mapwrite 0x017669 thru 0x021ffb (0xa993 bytes) 000291 1786401040.410659 read 0x0124e3 thru 0x013bb5 (0x16d3 bytes) 000293 1786401040.412239 mapread 0x030e0a thru 0x035a9f (0x4c96 bytes) 000294 1786401040.424647 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) 000295 1786401040.558562 MIRROR_EXTEND to 2 mirrors 000296 1786401041.095786 trunc from 0x03e8ad to 0x035e7d (0x8a2f bytes) 000297 1786401041.167147 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x35e7e): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35e7e): Mirror 2: 0x4408983b (stale) CRC-32 checksum value for chunk [0, 0x35e7e): Mirror 1: 0x4408983b Mirror 3: 0x4408983b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35e7e: skipped 000302 1786401041.899679 read 0x00042c thru 0x00a73e (0xa313 bytes) 000304 1786401041.916470 mapwrite 0x031eea thru 0x03bcfb (0x9e12 bytes) 000305 1786401042.116423 trunc from 0x03bcfc to 0x017642 (0x246b9 bytes) 000306 1786401042.148469 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x17643): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17643): Mirror 2: 0x669ad086 (stale) Mirror 3: 0x669ad086 (stale) CRC-32 checksum value for chunk [0, 0x17643): Mirror 1: 0x669ad086 Mirror 4: 0x669ad086 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17643: skipped 000307 1786401042.809201 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x17643): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17643): Mirror 2: 0x669ad086 (stale) Mirror 3: 0x669ad086 (stale) CRC-32 checksum value for chunk [0, 0x17643): Mirror 1: 0x669ad086 Mirror 4: 0x669ad086 Mirror 5: 0x669ad086 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17643: skipped 000308 1786401043.468973 write 0x00ad4e thru 0x01639f (0xb652 bytes) 000309 1786401043.544614 mapread 0x00501a thru 0x00ff4f (0xaf36 bytes) 000311 1786401043.561873 mapwrite 0x000565 thru 0x0089ca (0x8466 bytes) 000313 1786401043.607200 write 0x0185d1 thru 0x02538c (0xcdbc bytes) 000314 1786401043.609733 read 0x000b65 thru 0x009cfa (0x9196 bytes) 000315 1786401043.614974 MIRROR_RESYNC 5 mirrors 000316 1786401044.378679 trunc from 0x02538d to 0x00051b (0x24e71 bytes) 000317 1786401044.468777 mapread 0x000446 thru 0x00051b (0xd6 bytes) 000319 1786401044.495128 MIRROR_RESYNC 5 mirrors 000322 1786401045.141945 write 0x000a22 thru 0x00f6bb (0xec9a bytes) 000324 1786401045.208173 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0xf6bc): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf6bc): Mirror 2: 0x2465991e (stale) Mirror 3: 0x2465991e (stale) Mirror 4: 0x2465991e (stale) Mirror 5: 0x2465991e (stale) CRC-32 checksum value for chunk [0, 0xf6bc): Mirror 1: 0x657bd073 Mirror 6: 0x657bd073 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf6bc: skipped 000326 1786401046.017832 read 0x003014 thru 0x008db3 (0x5da0 bytes) 000327 1786401046.035508 write 0x01ef73 thru 0x02e0d1 (0xf15f bytes) 000328 1786401046.105224 trunc from 0x02e0d2 to 0x006721 (0x279b0 bytes) 000329 1786401046.126687 mapread 0x00453b thru 0x006721 (0x21e7 bytes) 000331 1786401046.148642 mapread 0x001a35 thru 0x006721 (0x4ced bytes) 000333 1786401046.154154 MIRROR_RESYNC 6 mirrors 000334 1786401046.929887 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) 000336 1786401047.200828 read 0x002816 thru 0x006721 (0x3f0c bytes) 000338 1786401047.212909 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) 000341 1786401047.489377 mapwrite 0x017d0f thru 0x02289f (0xab91 bytes) 000342 1786401047.644714 read 0x00377d thru 0x011b1a (0xe39e bytes) 000345 1786401047.655479 mapread 0x017216 thru 0x02289f (0xb68a bytes) 000346 1786401047.660407 MIRROR_RESYNC 4 mirrors 000349 1786401048.327643 read 0x00e7f0 thru 0x018c72 (0xa483 bytes) 000350 1786401048.346417 trunc from 0x0228a0 to 0x00699e (0x1bf01 bytes) 000351 1786401048.439251 read 0x004a06 thru 0x00699e (0x1f99 bytes) 000352 1786401048.450943 mapread 0x00432f thru 0x00699e (0x2670 bytes) 000353 1786401048.452215 mapread 0x0011c5 thru 0x00699e (0x57da bytes) 000354 1786401048.456651 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x699f): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x699f): Mirror 3: 0xf326fc1c (stale) Mirror 4: 0xf326fc1c (stale) Mirror 6: 0xf326fc1c (stale) CRC-32 checksum value for chunk [0, 0x699f): Mirror 1: 0xf326fc1c Mirror 7: 0xf326fc1c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x699f: skipped 000356 1786401049.001743 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x699f): Mirror 3: 0xf326fc1c (stale) Mirror 6: 0xf326fc1c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x699f): Mirror 3: 0xf326fc1c (stale) Mirror 6: 0xf326fc1c (stale) CRC-32 checksum value for chunk [0, 0x699f): Mirror 1: 0xf326fc1c Mirror 7: 0xf326fc1c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x699f: skipped 000357 1786401049.404137 write 0x008381 thru 0x01284b (0xa4cb bytes) 000362 1786401049.476487 mapwrite 0x01dad6 thru 0x025e5e (0x8389 bytes) 000363 1786401049.616607 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x25e5f): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25e5f): Mirror 3: 0x8ae8edc8 (stale) Mirror 6: 0x8ae8edc8 (stale) Mirror 7: 0xb731e48b (stale) CRC-32 checksum value for chunk [0, 0x25e5f): Mirror 1: 0x8b941ad0 Mirror 8: 0x8b941ad0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25e5f: skipped 000366 1786401050.268698 mapwrite 0x029d05 thru 0x0324eb (0x87e7 bytes) 000367 1786401050.479878 mapwrite 0x02fe33 thru 0x03bee9 (0xc0b7 bytes) 000368 1786401050.643834 mapwrite 0x015789 thru 0x024433 (0xecab bytes) 000369 1786401050.702134 read 0x016f38 thru 0x018c94 (0x1d5d bytes) 000370 1786401050.702925 read 0x02afac thru 0x030764 (0x57b9 bytes) 000373 1786401050.705314 mapwrite 0x006fff thru 0x00ed5d (0x7d5f bytes) 000376 1786401050.750326 MIRROR_RESYNC 5 mirrors 000377 1786401051.479406 write 0x035a44 thru 0x03ffff (0xa5bc bytes) 000380 1786401051.554992 trunc from 0x040000 to 0x00e797 (0x31868 bytes) 000381 1786401051.576938 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0xe798): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe798): Mirror 3: 0x8af944a2 (stale) Mirror 6: 0x8af944a2 (stale) Mirror 7: 0x8af944a2 (stale) Mirror 8: 0x8af944a2 (stale) CRC-32 checksum value for chunk [0, 0xe798): Mirror 1: 0x8af944a2 Mirror 9: 0x8af944a2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe798: skipped 000382 1786401052.303909 trunc from 0x00e798 to 0x003583 (0xb214 bytes) 000384 1786401052.397911 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3584): Mirror 6: 0xbbcd9302 (stale) Mirror 7: 0xbbcd9302 (stale) Mirror 8: 0xbbcd9302 (stale) Mirror 9: 0xbbcd9302 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3584): Mirror 6: 0xbbcd9302 (stale) Mirror 7: 0xbbcd9302 (stale) Mirror 8: 0xbbcd9302 (stale) Mirror 9: 0xbbcd9302 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3584: skipped 000385 1786401052.691290 read 0x00302b thru 0x003583 (0x559 bytes) 000387 1786401052.707791 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3584): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3584): Mirror 6: 0xbbcd9302 (stale) Mirror 7: 0xbbcd9302 (stale) Mirror 8: 0xbbcd9302 (stale) Mirror 9: 0xbbcd9302 (stale) CRC-32 checksum value for chunk [0, 0x3584): Mirror 1: 0xbbcd9302 Mirror 10: 0xbbcd9302 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3584: skipped 000388 1786401053.525164 trunc from 0x003584 to 0x00a696 (0x7113 bytes) 000389 1786401053.637136 read 0x008de5 thru 0x00a696 (0x18b2 bytes) 000390 1786401053.645361 mapread 0x0092d7 thru 0x00a696 (0x13c0 bytes) 000392 1786401053.648633 read 0x008684 thru 0x00a696 (0x2013 bytes) 000393 1786401053.649666 trunc from 0x00a697 to 0x00c81b (0x2185 bytes) 000394 1786401053.684638 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc81c): Mirror 7: 0x281cf33c (stale) Mirror 8: 0x281cf33c (stale) Mirror 9: 0x281cf33c (stale) Mirror 10: 0xf8edd490 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc81c): Mirror 7: 0x281cf33c (stale) Mirror 8: 0x281cf33c (stale) Mirror 9: 0x281cf33c (stale) Mirror 10: 0xf8edd490 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc81c: skipped 000397 1786401054.012804 mapwrite 0x0063bf thru 0x00f89a (0x94dc bytes) 000399 1786401054.107774 mapwrite 0x01c79f thru 0x024dcd (0x862f bytes) 000400 1786401054.179797 MIRROR_RESYNC 5 mirrors 000404 1786401055.016160 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) 000405 1786401055.303421 mapread 0x0210af thru 0x022c43 (0x1b95 bytes) 000406 1786401055.318887 mapread 0x002e16 thru 0x00abc1 (0x7dac bytes) 000407 1786401055.332768 mapwrite 0x008dcb thru 0x01494c (0xbb82 bytes) 000408 1786401055.523537 mapread 0x012ca2 thru 0x01be3b (0x919a bytes) 000410 1786401055.533917 trunc from 0x024dce to 0x032e9a (0xe0cd bytes) 000412 1786401055.575647 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x32e9b): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32e9b): Mirror 8: 0xc0bc338f (stale) Mirror 9: 0xc0bc338f (stale) Mirror 10: 0xc0bc338f (stale) CRC-32 checksum value for chunk [0, 0x32e9b): Mirror 1: 0x88d72e3c Mirror 11: 0x88d72e3c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32e9b: skipped truncating to largest ever: 0x3fd04 000413 1786401056.345517 trunc from 0x032e9b to 0x03fd03 (0xce69 bytes) 000415 1786401056.444015 trunc from 0x03fd04 to 0x03b371 (0x4992 bytes) 000418 1786401056.475552 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3b372): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b372): Mirror 8: 0xc909d838 (stale) Mirror 9: 0xc909d838 (stale) Mirror 10: 0xc909d838 (stale) Mirror 11: 0x7a81d88f (stale) CRC-32 checksum value for chunk [0, 0x3b372): Mirror 1: 0x7a81d88f Mirror 12: 0x7a81d88f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b372: skipped 000422 1786401057.434429 mapwrite 0x01199e thru 0x01ac29 (0x928c bytes) 000423 1786401057.580499 trunc from 0x03b372 to 0x02736b (0x14006 bytes) 000424 1786401057.611682 read 0x013836 thru 0x017587 (0x3d52 bytes) 000427 1786401057.613968 read 0x0019f4 thru 0x00ccbb (0xb2c8 bytes) 000428 1786401057.628901 mapwrite 0x000426 thru 0x00a44f (0xa02a bytes) 000429 1786401057.663138 mapwrite 0x029891 thru 0x029c7c (0x3ec bytes) 000430 1786401057.721504 write 0x0396c1 thru 0x03c41d (0x2d5d bytes) 000431 1786401057.723060 trunc from 0x03c41e to 0x027d17 (0x14706 bytes) 000432 1786401057.737545 mapread 0x01ced0 thru 0x025b6d (0x8c9e bytes) 000433 1786401057.738838 mapread 0x01a382 thru 0x0207f5 (0x6474 bytes) 000435 1786401057.740826 read 0x0176dd thru 0x01ed47 (0x766b bytes) 000437 1786401057.746359 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27d18): Mirror 9: 0xab8cc6e7 (stale) Mirror 10: 0xab8cc6e7 (stale) Mirror 11: 0xdf07658a (stale) Mirror 12: 0xdf07658a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27d18): Mirror 9: 0xab8cc6e7 (stale) Mirror 10: 0xab8cc6e7 (stale) Mirror 11: 0xdf07658a (stale) Mirror 12: 0xdf07658a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27d18: skipped 000441 1786401058.128980 read 0x00579d thru 0x006fdc (0x1840 bytes) 000442 1786401058.135268 read 0x024e02 thru 0x027d17 (0x2f16 bytes) 000443 1786401058.144851 read 0x003909 thru 0x0080b9 (0x47b1 bytes) 000444 1786401058.145624 mapread 0x019fea thru 0x026509 (0xc520 bytes) 000445 1786401058.147311 trunc from 0x027d18 to 0x0264c0 (0x1857 bytes) 000446 1786401058.163732 read 0x001326 thru 0x00212c (0xe07 bytes) 000448 1786401058.165687 read 0x021b50 thru 0x0264c0 (0x4971 bytes) 000449 1786401058.166908 trunc from 0x0264c1 to 0x032136 (0xbc76 bytes) 000450 1786401058.185817 read 0x0190b5 thru 0x022ead (0x9df9 bytes) 000451 1786401058.186812 mapwrite 0x03aacf thru 0x03ffff (0x5531 bytes) 000452 1786401058.250754 mapwrite 0x03451e thru 0x03ecd7 (0xa7ba bytes) 000453 1786401058.298301 write 0x012981 thru 0x019e7a (0x74fa bytes) 000454 1786401058.302953 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x754b9cb (stale) Mirror 10: 0x754b9cb (stale) Mirror 11: 0x4fbe792a (stale) Mirror 12: 0x4fbe792a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd4d00f17 Mirror 13: 0xd4d00f17 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000455 1786401058.903731 read 0x0215d0 thru 0x02fc78 (0xe6a9 bytes) 000456 1786401058.927211 mapwrite 0x0024e6 thru 0x00942b (0x6f46 bytes) 000459 1786401059.087929 mapread 0x0113a9 thru 0x01cd86 (0xb9de bytes) 000461 1786401059.112799 mapwrite 0x02d374 thru 0x039436 (0xc0c3 bytes) 000466 1786401059.194548 mapread 0x00d632 thru 0x0116fc (0x40cb bytes) 000467 1786401059.196358 mapwrite 0x00705f thru 0x00d062 (0x6004 bytes) 000468 1786401059.322446 trunc from 0x040000 to 0x02fc94 (0x1036b bytes) 000470 1786401059.380369 trunc from 0x02fc95 to 0x01ab24 (0x15170 bytes) 000471 1786401059.394431 mapread 0x011986 thru 0x01ab24 (0x919f bytes) 000472 1786401059.395688 mapwrite 0x001644 thru 0x00e135 (0xcaf2 bytes) 000479 1786401059.435916 mapread 0x011de6 thru 0x014c65 (0x2e80 bytes) 000481 1786401059.437789 mapread 0x00421e thru 0x00db1f (0x9902 bytes) 000484 1786401059.440069 read 0x01960b thru 0x01ab24 (0x151a bytes) 000485 1786401059.444425 MIRROR_RESYNC 6 mirrors 000486 1786401060.307514 MIRROR_EXTEND to 7 mirrors 000487 1786401060.932620 trunc from 0x01ab25 to 0x003642 (0x174e2 bytes) 000488 1786401061.061390 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3643): Mirror 10: 0x9f4d37b7 (stale) Mirror 11: 0x9f4d37b7 (stale) Mirror 12: 0x9f4d37b7 (stale) Mirror 13: 0x9f4d37b7 (stale) Mirror 14: 0x9f4d37b7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3643): Mirror 10: 0x9f4d37b7 (stale) Mirror 11: 0x9f4d37b7 (stale) Mirror 12: 0x9f4d37b7 (stale) Mirror 13: 0x9f4d37b7 (stale) Mirror 14: 0x9f4d37b7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3643: skipped 000491 1786401061.416009 trunc from 0x003643 to 0x03d285 (0x39c43 bytes) 000496 1786401061.456055 mapread 0x00c323 thru 0x010e48 (0x4b26 bytes) 000499 1786401061.521937 write 0x027f68 thru 0x033e9f (0xbf38 bytes) 000503 1786401061.539763 mapwrite 0x01cbb2 thru 0x024100 (0x754f bytes) 000505 1786401061.600307 trunc from 0x03d286 to 0x03992a (0x395b bytes) 000507 1786401061.630593 read 0x0214c1 thru 0x025a6d (0x45ad bytes) 000509 1786401061.639587 read 0x003a3c thru 0x00fbcf (0xc194 bytes) 000513 1786401061.663640 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3992b): Mirror 10: 0x480d58b9 (stale) Mirror 11: 0x480d58b9 (stale) Mirror 12: 0x480d58b9 (stale) Mirror 14: 0x480d58b9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3992b): Mirror 10: 0x480d58b9 (stale) Mirror 11: 0x480d58b9 (stale) Mirror 12: 0x480d58b9 (stale) Mirror 14: 0x480d58b9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3992b: skipped 000514 1786401062.112972 trunc from 0x03992b to 0x0345f1 (0x5339 bytes) 000515 1786401062.125973 mapwrite 0x024d7a thru 0x02e7c8 (0x9a4f bytes) 000518 1786401062.195630 write 0x02c751 thru 0x032df0 (0x66a0 bytes) 000519 1786401062.200893 mapread 0x003b48 thru 0x003dd5 (0x28e bytes) 000520 1786401062.209446 mapread 0x0208f6 thru 0x029c15 (0x9320 bytes) 000521 1786401062.211314 write 0x02254d thru 0x023c0b (0x16bf bytes) 000522 1786401062.212484 read 0x01273d thru 0x01f7ef (0xd0b3 bytes) 000523 1786401062.225341 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x345f2): Mirror 10: 0xf78219a2 (stale) Mirror 11: 0xf78219a2 (stale) Mirror 14: 0xf78219a2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x345f2): Mirror 10: 0xf78219a2 (stale) Mirror 11: 0xf78219a2 (stale) Mirror 14: 0xf78219a2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x345f2: skipped 000527 1786401062.553442 write 0x0205a7 thru 0x0279b3 (0x740d bytes) 000528 1786401062.570861 MIRROR_RESYNC 4 mirrors 000529 1786401063.339003 trunc from 0x0345f2 to 0x0030c6 (0x3152b bytes) 000530 1786401063.404860 write 0x0109f1 thru 0x01e24b (0xd85b bytes) 000532 1786401063.408520 trunc from 0x01e24c to 0x0248ed (0x66a2 bytes) 000536 1786401063.437024 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x248ee): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x248ee): Mirror 10: 0xf27c64bd (stale) Mirror 11: 0xf27c64bd (stale) Mirror 14: 0xf27c64bd (stale) CRC-32 checksum value for chunk [0, 0x248ee): Mirror 1: 0x7ef9cc16 Mirror 15: 0x7ef9cc16 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x248ee: skipped 000537 1786401064.074748 write 0x02592e thru 0x030f2d (0xb600 bytes) 000538 1786401064.147435 mapwrite 0x00cfc3 thru 0x01ac03 (0xdc41 bytes) 000539 1786401064.226972 mapread 0x00b364 thru 0x0145ad (0x924a bytes) 000540 1786401064.228504 trunc from 0x030f2e to 0x01116e (0x1fdbf bytes) 000541 1786401064.254970 mapwrite 0x00cba2 thru 0x017861 (0xacc0 bytes) 000542 1786401064.323485 write 0x03b3c4 thru 0x03c259 (0xe96 bytes) 000543 1786401064.324716 trunc from 0x03c25a to 0x034886 (0x79d3 bytes) 000545 1786401064.349775 write 0x020fb2 thru 0x021f3b (0xf8a bytes) 000547 1786401064.358894 mapread 0x01abe7 thru 0x01c563 (0x197d bytes) 000548 1786401064.372361 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x34887): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34887): Mirror 10: 0x73ac1a40 (stale) Mirror 11: 0x73ac1a40 (stale) Mirror 14: 0x73ac1a40 (stale) Mirror 15: 0xfbfbf89a (stale) CRC-32 checksum value for chunk [0, 0x34887): Mirror 1: 0xf7d56017 Mirror 16: 0xf7d56017 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34887: skipped 000551 1786401065.063766 mapread 0x02d05d thru 0x02d82d (0x7d1 bytes) 000552 1786401065.076703 write 0x000a9d thru 0x00be46 (0xb3aa bytes) 000556 1786401065.171755 mapread 0x01c3dd thru 0x028808 (0xc42c bytes) 000557 1786401065.205364 write 0x011102 thru 0x01ed7a (0xdc79 bytes) 000559 1786401065.213575 mapwrite 0x026bc4 thru 0x035847 (0xec84 bytes) 000560 1786401065.336280 mapwrite 0x0199da thru 0x01e011 (0x4638 bytes) 000561 1786401065.386044 write 0x002092 thru 0x00e759 (0xc6c8 bytes) 000562 1786401065.400951 MIRROR_RESYNC 6 mirrors 000563 1786401066.529623 mapread 0x0065ae thru 0x0076e8 (0x113b bytes) 000565 1786401066.539929 trunc from 0x035848 to 0x00c4fe (0x29349 bytes) 000567 1786401066.624732 MIRROR_RESYNC 6 mirrors 000568 1786401067.638512 read 0x005090 thru 0x00b97e (0x68ef bytes) 000570 1786401067.648258 mapread 0x007ca6 thru 0x00c4fe (0x4859 bytes) 000571 1786401067.653297 mapwrite 0x03cfdd thru 0x03d827 (0x84b bytes) 000574 1786401067.782869 mapwrite 0x03a837 thru 0x03ffff (0x57c9 bytes) 000575 1786401067.836758 write 0x03c8e7 thru 0x03ffff (0x3719 bytes) 000576 1786401067.838552 mapread 0x0080b3 thru 0x00ffd4 (0x7f22 bytes) 000577 1786401067.848980 mapwrite 0x036f7f thru 0x03c8aa (0x592c bytes) 000580 1786401067.890916 read 0x035352 thru 0x03bd74 (0x6a23 bytes) 000581 1786401067.901658 mapread 0x039e76 thru 0x03e5db (0x4766 bytes) 000582 1786401067.905930 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xeb5a2ad3 (stale) Mirror 11: 0xeb5a2ad3 (stale) Mirror 14: 0xeb5a2ad3 (stale) Mirror 15: 0xeb5a2ad3 (stale) Mirror 16: 0xeb5a2ad3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4f17883 Mirror 17: 0x4f17883 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000584 1786401068.710522 read 0x038b59 thru 0x03ffff (0x74a7 bytes) 000586 1786401068.731838 read 0x00e979 thru 0x01bab1 (0xd139 bytes) 000587 1786401068.746866 write 0x0126b1 thru 0x01d0d7 (0xaa27 bytes) 000588 1786401068.825382 trunc from 0x040000 to 0x0087ce (0x37831 bytes) 000590 1786401068.848555 MIRROR_RESYNC 7 mirrors 000591 1786401070.010944 read 0x004a58 thru 0x006486 (0x1a2f bytes) 000593 1786401070.019923 write 0x0163ed thru 0x019804 (0x3418 bytes) 000594 1786401070.084232 MIRROR_RESYNC 7 mirrors 000595 1786401070.940764 trunc from 0x019805 to 0x008738 (0x110cc bytes) 000598 1786401071.035023 mapwrite 0x03ec37 thru 0x03ffff (0x13c9 bytes) 000600 1786401071.129948 MIRROR_RESYNC 7 mirrors 000601 1786401072.188786 mapread 0x022ffe thru 0x02bc0b (0x8c0e bytes) 000603 1786401072.214575 MIRROR_EXTEND to 8 mirrors 000604 1786401072.832837 mapwrite 0x03e7a6 thru 0x03ffff (0x185a bytes) 000605 1786401073.041324 read 0x000d1a thru 0x00cf5c (0xc243 bytes) 000607 1786401073.063218 mapread 0x01a83d thru 0x023fab (0x976f bytes) 000608 1786401073.097619 mapread 0x01584c thru 0x01fd83 (0xa538 bytes) 000610 1786401073.101003 read 0x02074d thru 0x02f3f7 (0xecab bytes) 000612 1786401073.115582 MIRROR_RESYNC 8 mirrors 000613 1786401074.105015 write 0x01d42c thru 0x01ea72 (0x1647 bytes) 000614 1786401074.162365 trunc from 0x040000 to 0x02663d (0x199c2 bytes) 000615 1786401074.178093 mapread 0x0115ab thru 0x012536 (0xf8c bytes) 000618 1786401074.191657 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2663e): Mirror 10: 0xa961bbca (stale) Mirror 11: 0xa961bbca (stale) Mirror 14: 0xa961bbca (stale) Mirror 15: 0xa961bbca (stale) Mirror 16: 0xa961bbca (stale) Mirror 18: 0xa961bbca (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2663e): Mirror 10: 0xa961bbca (stale) Mirror 11: 0xa961bbca (stale) Mirror 14: 0xa961bbca (stale) Mirror 15: 0xa961bbca (stale) Mirror 16: 0xa961bbca (stale) Mirror 18: 0xa961bbca (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2663e: skipped 000619 1786401074.744440 write 0x0097dd thru 0x018332 (0xeb56 bytes) 000621 1786401074.754500 write 0x01f504 thru 0x029e3f (0xa93c bytes) 000623 1786401074.760584 read 0x01dd53 thru 0x01f8ae (0x1b5c bytes) 000624 1786401074.768531 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29e40): Mirror 10: 0x5383ef8e (stale) Mirror 14: 0x5383ef8e (stale) Mirror 15: 0x5383ef8e (stale) Mirror 16: 0x5383ef8e (stale) Mirror 18: 0x5383ef8e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29e40): Mirror 10: 0x5383ef8e (stale) Mirror 14: 0x5383ef8e (stale) Mirror 15: 0x5383ef8e (stale) Mirror 16: 0x5383ef8e (stale) Mirror 18: 0x5383ef8e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29e40: skipped 000626 1786401075.147165 mapread 0x025f28 thru 0x029e3f (0x3f18 bytes) 000628 1786401075.171140 trunc from 0x029e40 to 0x018363 (0x11adc bytes) 000629 1786401075.189865 write 0x02a466 thru 0x033838 (0x93d3 bytes) 000630 1786401075.192165 read 0x019881 thru 0x022736 (0x8eb6 bytes) 000632 1786401075.202215 write 0x02ee91 thru 0x03ca0b (0xdb7b bytes) 000635 1786401075.211646 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ca0c): Mirror 14: 0xb8b3756 (stale) Mirror 15: 0xb8b3756 (stale) Mirror 16: 0xb8b3756 (stale) Mirror 18: 0xb8b3756 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ca0c): Mirror 14: 0xb8b3756 (stale) Mirror 15: 0xb8b3756 (stale) Mirror 16: 0xb8b3756 (stale) Mirror 18: 0xb8b3756 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ca0c: skipped 000641 1786401075.620310 trunc from 0x03ca0c to 0x036778 (0x6293 bytes) 000646 1786401075.643543 mapwrite 0x026b76 thru 0x0342e7 (0xd772 bytes) 000647 1786401075.736426 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x36779): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36779): Mirror 14: 0xe67c1036 (stale) Mirror 15: 0xe67c1036 (stale) Mirror 16: 0xe67c1036 (stale) Mirror 18: 0xe67c1036 (stale) CRC-32 checksum value for chunk [0, 0x36779): Mirror 1: 0x811c2ed5 Mirror 19: 0x811c2ed5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36779: skipped 000648 1786401076.459290 mapwrite 0x01f317 thru 0x02384a (0x4534 bytes) 000649 1786401076.672852 write 0x02c290 thru 0x02dd69 (0x1ada bytes) 000650 1786401076.695382 read 0x001546 thru 0x00bf58 (0xaa13 bytes) 000651 1786401076.712774 write 0x03e3cf thru 0x03ffff (0x1c31 bytes) 000653 1786401076.726591 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0xf5254f6 (stale) Mirror 15: 0xf5254f6 (stale) Mirror 16: 0xf5254f6 (stale) Mirror 19: 0x60593c20 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0xf5254f6 (stale) Mirror 15: 0xf5254f6 (stale) Mirror 16: 0xf5254f6 (stale) Mirror 19: 0x60593c20 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000654 1786401077.069341 mapwrite 0x02f33b thru 0x03c1ad (0xce73 bytes) 000655 1786401077.148894 mapwrite 0x008cc5 thru 0x0143f7 (0xb733 bytes) 000656 1786401077.225861 trunc from 0x040000 to 0x00f401 (0x30bfe bytes) 000660 1786401077.255220 mapread 0x00954c thru 0x00f401 (0x5eb6 bytes) 000662 1786401077.257223 trunc from 0x00f402 to 0x02acec (0x1b8eb bytes) 000664 1786401077.297869 MIRROR_RESYNC 5 mirrors 000665 1786401078.405935 read 0x017516 thru 0x023347 (0xbe32 bytes) 000666 1786401078.414828 mapread 0x0253d2 thru 0x02acec (0x591b bytes) 000667 1786401078.432289 mapread 0x0286b7 thru 0x02acec (0x2636 bytes) 000673 1786401078.435212 mapread 0x0070b3 thru 0x007894 (0x7e2 bytes) 000675 1786401078.451514 MIRROR_EXTEND to 6 mirrors 000676 1786401079.070209 write 0x03e6ec thru 0x03ffff (0x1914 bytes) 000678 1786401079.129709 write 0x012f6b thru 0x01ef97 (0xc02d bytes) 000681 1786401079.142778 mapwrite 0x00368b thru 0x00dbbf (0xa535 bytes) 000684 1786401079.219417 trunc from 0x040000 to 0x01c37e (0x23c81 bytes) 000685 1786401079.285392 read 0x00bb27 thru 0x00fb95 (0x406f bytes) 000687 1786401079.287967 mapwrite 0x03ff3e thru 0x03ffff (0xc2 bytes) 000688 1786401079.365588 mapread 0x005adc thru 0x01242f (0xc954 bytes) 000690 1786401079.373537 trunc from 0x040000 to 0x0025ee (0x3da11 bytes) 000691 1786401079.457814 write 0x025f15 thru 0x02ff57 (0xa043 bytes) 000692 1786401079.461336 mapread 0x001af3 thru 0x010b46 (0xf054 bytes) 000693 1786401079.470844 trunc from 0x02ff58 to 0x023ea1 (0xc0b6 bytes) 000696 1786401079.550893 mapwrite 0x02f3fd thru 0x035345 (0x5f49 bytes) 000697 1786401079.644873 trunc from 0x035346 to 0x030fab (0x439a bytes) 000698 1786401079.667508 mapwrite 0x003993 thru 0x010c9f (0xd30d bytes) 000699 1786401079.730965 mapread 0x026f09 thru 0x0291e7 (0x22df bytes) 000700 1786401079.745282 trunc from 0x030fac to 0x00e53e (0x22a6d bytes) 000702 1786401079.763322 mapread 0x006364 thru 0x007f8f (0x1c2c bytes) 000703 1786401079.764766 trunc from 0x00e53f to 0x03ba43 (0x2d505 bytes) 000705 1786401079.783262 mapwrite 0x03fc00 thru 0x03ffff (0x400 bytes) 000706 1786401079.839527 write 0x02668a thru 0x02af4b (0x48c2 bytes) 000707 1786401079.848903 mapread 0x034f02 thru 0x03f785 (0xa884 bytes) 000708 1786401079.858290 write 0x00d2ff thru 0x01659b (0x929d bytes) 000709 1786401079.866261 write 0x01e7db thru 0x020cfc (0x2522 bytes) 000710 1786401079.879522 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0xf950c5ef (stale) Mirror 15: 0xf950c5ef (stale) Mirror 16: 0xf950c5ef (stale) Mirror 19: 0xf950c5ef (stale) Mirror 20: 0xf950c5ef (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xaa238e8 Mirror 21: 0xaa238e8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000711 1786401080.892247 write 0x00b2b5 thru 0x012e36 (0x7b82 bytes) 000716 1786401080.964819 read 0x00dd10 thru 0x01dccc (0xffbd bytes) 000717 1786401080.975265 write 0x028876 thru 0x02d89e (0x5029 bytes) 000718 1786401080.990831 read 0x0210cd thru 0x02eac5 (0xd9f9 bytes) 000719 1786401081.001731 write 0x0047d0 thru 0x00c377 (0x7ba8 bytes) 000720 1786401081.003467 mapread 0x006d8d thru 0x007303 (0x577 bytes) 000721 1786401081.004837 write 0x02fbb2 thru 0x03c4b2 (0xc901 bytes) 000723 1786401081.007582 mapwrite 0x026ea1 thru 0x0325a5 (0xb705 bytes) 000724 1786401081.056755 read 0x006cd8 thru 0x012f95 (0xc2be bytes) 000726 1786401081.058949 mapwrite 0x00deb9 thru 0x015cff (0x7e47 bytes) 000727 1786401081.097995 trunc from 0x040000 to 0x03511c (0xaee3 bytes) 000728 1786401081.128285 read 0x017d73 thru 0x026521 (0xe7af bytes) 000732 1786401081.135417 MIRROR_RESYNC 7 mirrors 000733 1786401082.094790 mapread 0x02ff88 thru 0x03511c (0x5195 bytes) 000734 1786401082.111882 mapread 0x030e1c thru 0x03132a (0x50f bytes) 000735 1786401082.112800 mapread 0x0286ec thru 0x03511c (0xca31 bytes) 000737 1786401082.123532 mapwrite 0x02eca4 thru 0x031708 (0x2a65 bytes) 000743 1786401082.275165 trunc from 0x03511d to 0x0099c9 (0x2b753 bytes) 000744 1786401082.313826 mapread 0x004868 thru 0x0099c9 (0x5162 bytes) 000747 1786401082.326857 mapread 0x004667 thru 0x0099c9 (0x5363 bytes) 000748 1786401082.328424 trunc from 0x0099ca to 0x023ad3 (0x1a10a bytes) 000750 1786401082.342146 write 0x00668c thru 0x00ad00 (0x4675 bytes) 000753 1786401082.352400 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x23ad4): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23ad4): Mirror 14: 0x6f2212bd (stale) Mirror 15: 0x6f2212bd (stale) Mirror 16: 0x6f2212bd (stale) Mirror 19: 0x6f2212bd (stale) Mirror 20: 0x6f2212bd (stale) Mirror 21: 0x6f2212bd (stale) CRC-32 checksum value for chunk [0, 0x23ad4): Mirror 1: 0x3a022fc3 Mirror 22: 0x3a022fc3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23ad4: skipped 000755 1786401083.198795 MIRROR_RESYNC 8 mirrors 000758 1786401084.254314 write 0x02ed2b thru 0x03adc0 (0xc096 bytes) 000759 1786401084.327868 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3adc1): Mirror 14: 0xedc43c4b (stale) Mirror 15: 0xedc43c4b (stale) Mirror 19: 0xedc43c4b (stale) Mirror 20: 0xedc43c4b (stale) Mirror 21: 0xedc43c4b (stale) Mirror 22: 0xedc43c4b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3adc1): Mirror 14: 0xedc43c4b (stale) Mirror 15: 0xedc43c4b (stale) Mirror 19: 0xedc43c4b (stale) Mirror 20: 0xedc43c4b (stale) Mirror 21: 0xedc43c4b (stale) Mirror 22: 0xedc43c4b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3adc1: skipped 000760 1786401084.924688 mapread 0x033402 thru 0x03709d (0x3c9c bytes) 000762 1786401084.943775 read 0x0157b2 thru 0x023ab2 (0xe301 bytes) 000763 1786401084.951888 trunc from 0x03adc1 to 0x018fe6 (0x21dda bytes) 000764 1786401084.968306 write 0x031887 thru 0x03d0cc (0xb846 bytes) 000766 1786401084.971804 read 0x035e1d thru 0x037bd9 (0x1dbd bytes) 000767 1786401084.972719 read 0x00f676 thru 0x018ab5 (0x9440 bytes) 000768 1786401084.991697 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3d0cd): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d0cd): Mirror 14: 0xa4e4b2ae (stale) Mirror 15: 0xa4e4b2ae (stale) Mirror 19: 0xa4e4b2ae (stale) Mirror 20: 0xa4e4b2ae (stale) Mirror 21: 0xa4e4b2ae (stale) Mirror 22: 0xa4e4b2ae (stale) CRC-32 checksum value for chunk [0, 0x3d0cd): Mirror 1: 0x4b27929b Mirror 23: 0x4b27929b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d0cd: skipped 000769 1786401085.830921 mapwrite 0x014767 thru 0x017592 (0x2e2c bytes) 000770 1786401085.966859 MIRROR_RESYNC 8 mirrors 000772 1786401087.090664 MIRROR_RESYNC 8 mirrors 000773 1786401087.758488 read 0x00032e thru 0x001440 (0x1113 bytes) 000776 1786401087.766021 trunc from 0x03d0cd to 0x01934c (0x23d80 bytes) 000778 1786401087.824271 mapread 0x000cc7 thru 0x00bfcb (0xb305 bytes) 000780 1786401087.848258 trunc from 0x01934d to 0x03a2af (0x20f63 bytes) 000781 1786401087.870761 read 0x005344 thru 0x00c3d5 (0x7092 bytes) 000784 1786401087.873835 write 0x00a7bc thru 0x00e46a (0x3caf bytes) 000785 1786401087.875295 write 0x02c3f9 thru 0x03be60 (0xfa68 bytes) 000786 1786401087.885618 mapwrite 0x03e4a7 thru 0x03ffff (0x1b59 bytes) 000788 1786401087.944845 mapwrite 0x02a844 thru 0x02b973 (0x1130 bytes) 000790 1786401087.972986 write 0x035123 thru 0x03b79d (0x667b bytes) 000792 1786401087.975072 write 0x0301b6 thru 0x033128 (0x2f73 bytes) 000793 1786401087.976381 mapwrite 0x023480 thru 0x0321d9 (0xed5a bytes) 000800 1786401088.032616 mapwrite 0x034b5a thru 0x03ffff (0xb4a6 bytes) 000802 1786401088.080674 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xd59f79f3 (stale) Mirror 19: 0xd59f79f3 (stale) Mirror 20: 0xd59f79f3 (stale) Mirror 21: 0xd59f79f3 (stale) Mirror 22: 0xd59f79f3 (stale) Mirror 23: 0xd59f79f3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xd59f79f3 (stale) Mirror 19: 0xd59f79f3 (stale) Mirror 20: 0xd59f79f3 (stale) Mirror 21: 0xd59f79f3 (stale) Mirror 22: 0xd59f79f3 (stale) Mirror 23: 0xd59f79f3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000803 1786401088.781739 trunc from 0x040000 to 0x0036cc (0x3c933 bytes) 000804 1786401088.797250 read 0x002f61 thru 0x0036cc (0x76c bytes) 000805 1786401088.804340 trunc from 0x0036cd to 0x011252 (0xdb86 bytes) 000806 1786401088.818804 read 0x000e3f thru 0x002811 (0x19d3 bytes) 000807 1786401088.829778 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x11253): Mirror 15: 0x81e1477 (stale) Mirror 19: 0x81e1477 (stale) Mirror 21: 0x81e1477 (stale) Mirror 22: 0x81e1477 (stale) Mirror 23: 0x81e1477 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11253): Mirror 15: 0x81e1477 (stale) Mirror 19: 0x81e1477 (stale) Mirror 21: 0x81e1477 (stale) Mirror 22: 0x81e1477 (stale) Mirror 23: 0x81e1477 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11253: skipped 000808 1786401089.223833 MIRROR_RESYNC 6 mirrors 000810 1786401090.227503 read 0x005e7b thru 0x00c742 (0x68c8 bytes) 000811 1786401090.236903 mapread 0x00e3d8 thru 0x011252 (0x2e7b bytes) 000812 1786401090.255764 trunc from 0x011253 to 0x00a022 (0x7230 bytes) 000813 1786401090.334288 read 0x009942 thru 0x00a022 (0x6e1 bytes) 000814 1786401090.345752 trunc from 0x00a023 to 0x039a60 (0x2fa3e bytes) 000816 1786401090.364957 read 0x0369b4 thru 0x037032 (0x67f bytes) 000817 1786401090.378813 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39a61): Mirror 19: 0xad518da8 (stale) Mirror 21: 0xad518da8 (stale) Mirror 22: 0xad518da8 (stale) Mirror 23: 0xad518da8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39a61): Mirror 19: 0xad518da8 (stale) Mirror 21: 0xad518da8 (stale) Mirror 22: 0xad518da8 (stale) Mirror 23: 0xad518da8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39a61: skipped 000818 1786401090.647367 read 0x01649e thru 0x025264 (0xedc7 bytes) 000820 1786401090.659890 trunc from 0x039a61 to 0x01695b (0x23105 bytes) 000821 1786401090.682866 MIRROR_RESYNC 5 mirrors 000824 1786401091.472692 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) 000826 1786401091.773168 read 0x00a2d1 thru 0x01695b (0xc68b bytes) 000831 1786401091.791759 read 0x00ad25 thru 0x0153d8 (0xa6b4 bytes) 000832 1786401091.792590 read 0x00def3 thru 0x01695b (0x8a69 bytes) 000834 1786401091.793615 write 0x013b71 thru 0x022479 (0xe909 bytes) 000835 1786401091.934885 read 0x0101a0 thru 0x017260 (0x70c1 bytes) 000837 1786401091.937221 read 0x0020cb thru 0x011961 (0xf897 bytes) 000838 1786401091.938637 trunc from 0x02247a to 0x03fad0 (0x1d657 bytes) 000839 1786401091.958210 read 0x01628c thru 0x01f2ba (0x902f bytes) 000840 1786401091.959584 mapwrite 0x028429 thru 0x036e12 (0xe9ea bytes) 000842 1786401092.037397 read 0x0263ed thru 0x0315b3 (0xb1c7 bytes) 000843 1786401092.038417 read 0x023293 thru 0x025d07 (0x2a75 bytes) 000844 1786401092.039152 read 0x029ef3 thru 0x0369f7 (0xcb05 bytes) 000846 1786401092.040695 mapread 0x01a1c3 thru 0x01a906 (0x744 bytes) 000848 1786401092.046196 MIRROR_RESYNC 4 mirrors 000849 1786401092.935704 write 0x003997 thru 0x007cff (0x4369 bytes) 000850 1786401092.996382 MIRROR_RESYNC 4 mirrors 000853 1786401093.883307 mapwrite 0x038daa thru 0x039f5d (0x11b4 bytes) 000854 1786401094.050073 mapread 0x03edcb thru 0x03fad0 (0xd06 bytes) 000855 1786401094.051878 write 0x02389f thru 0x02b744 (0x7ea6 bytes) 000856 1786401094.071674 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fad1): Mirror 19: 0x3c6dc0e5 (stale) Mirror 23: 0x3c6dc0e5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fad1): Mirror 19: 0x3c6dc0e5 (stale) Mirror 23: 0x3c6dc0e5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fad1: skipped 000857 1786401094.362519 MIRROR_RESYNC 3 mirrors 000858 1786401095.032296 trunc from 0x03fad1 to 0x002747 (0x3d389 bytes) 000861 1786401095.286666 MIRROR_RESYNC 3 mirrors 000862 1786401096.088709 mapwrite 0x006078 thru 0x01012e (0xa0b7 bytes) 000863 1786401096.266145 write 0x00868e thru 0x013f14 (0xb887 bytes) 000864 1786401096.268884 write 0x02cca0 thru 0x03c26f (0xf5d0 bytes) 000866 1786401096.273880 trunc from 0x03c270 to 0x0231fc (0x19073 bytes) 000867 1786401096.309746 mapread 0x017213 thru 0x01b585 (0x4373 bytes) 000869 1786401096.321366 read 0x01f192 thru 0x0231fc (0x406b bytes) 000870 1786401096.330429 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x231fd): Mirror 23: 0xa2107f2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x231fd): Mirror 23: 0xa2107f2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x231fd: skipped 000872 1786401096.798175 MIRROR_RESYNC 2 mirrors 000874 1786401097.438836 read 0x0087a5 thru 0x01747d (0xecd9 bytes) 000875 1786401097.467355 MIRROR_RESYNC 2 mirrors 000876 1786401097.750790 mapread 0x011911 thru 0x013b5d (0x224d bytes) 000878 1786401097.751908 mapwrite 0x01641b thru 0x025a0e (0xf5f4 bytes) 000881 1786401097.880614 write 0x014fab thru 0x01bfac (0x7002 bytes) 000882 1786401097.882174 write 0x0155e2 thru 0x01ed11 (0x9730 bytes) 000883 1786401097.887897 MIRROR_RESYNC 2 mirrors 000884 1786401098.422473 mapread 0x00c511 thru 0x0125da (0x60ca bytes) 000886 1786401098.440845 mapread 0x014b70 thru 0x01fcfd (0xb18e bytes) 000887 1786401098.451434 trunc from 0x025a0f to 0x024ae3 (0xf2b bytes) 000890 1786401098.515146 mapwrite 0x0114b3 thru 0x01cb9e (0xb6ec bytes) 000891 1786401098.572923 trunc from 0x024ae4 to 0x02f1ac (0xa6c9 bytes) 000897 1786401098.594864 trunc from 0x02f1ad to 0x00a433 (0x24d79 bytes) 000898 1786401098.615162 mapwrite 0x026e0e thru 0x031c41 (0xae34 bytes) 000903 1786401098.697659 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x31c42): Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31c42): Mirror 23: 0x518a6261 (stale) CRC-32 checksum value for chunk [0, 0x31c42): Mirror 1: 0x700eb4d7 Mirror 24: 0x700eb4d7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31c42: skipped 000904 1786401099.253809 read 0x000cec thru 0x00cf8f (0xc2a4 bytes) 000905 1786401099.265740 trunc from 0x031c42 to 0x0175d9 (0x1a668 bytes) 000906 1786401099.375564 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x175da): Mirror 24: 0x373e0140 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x175da): Mirror 24: 0x373e0140 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x175da: skipped 000907 1786401099.603580 mapread 0x0051c0 thru 0x005368 (0x1a9 bytes) 000909 1786401099.610862 read 0x00891b thru 0x0175d9 (0xecbf bytes) 000910 1786401099.620442 mapread 0x007827 thru 0x00bb75 (0x434f bytes) 000912 1786401099.626301 read 0x00fa56 thru 0x0121ff (0x27aa bytes) 000913 1786401099.627139 write 0x02d613 thru 0x03a29b (0xcc89 bytes) 000914 1786401099.633498 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3a29c): Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a29c): Mirror 24: 0x5c0cad67 (stale) CRC-32 checksum value for chunk [0, 0x3a29c): Mirror 1: 0x1850f02f Mirror 25: 0x1850f02f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a29c: skipped 000915 1786401100.431350 mapwrite 0x00a1c1 thru 0x010e52 (0x6c92 bytes) 000916 1786401100.631849 trunc from 0x03a29c to 0x003d50 (0x3654b bytes) 000918 1786401100.687716 MIRROR_RESYNC 3 mirrors 000919 1786401101.312378 mapwrite 0x013de6 thru 0x023b84 (0xfd9f bytes) 000920 1786401101.480792 MIRROR_RESYNC 3 mirrors 000923 1786401102.015394 read 0x00d40e thru 0x00dd5f (0x952 bytes) 000924 1786401102.024696 MIRROR_EXTEND to 4 mirrors 000925 1786401102.554992 write 0x034f4c thru 0x03a221 (0x52d6 bytes) 000926 1786401102.607557 write 0x03c80a thru 0x03ffff (0x37f6 bytes) 000929 1786401102.610955 write 0x029ddf thru 0x034916 (0xab38 bytes) 000930 1786401102.616454 write 0x027666 thru 0x02c550 (0x4eeb bytes) 000935 1786401102.626669 write 0x02b7cd thru 0x030aad (0x52e1 bytes) 000938 1786401102.629690 write 0x006b99 thru 0x00a4a5 (0x390d bytes) 000939 1786401102.647643 MIRROR_RESYNC 4 mirrors 000940 1786401103.404605 MIRROR_RESYNC 4 mirrors 000941 1786401103.734367 mapread 0x0208d5 thru 0x02d973 (0xd09f bytes) 000943 1786401103.766640 trunc from 0x040000 to 0x01980a (0x267f5 bytes) 000944 1786401103.825486 trunc from 0x01980b to 0x031c88 (0x1847e bytes) 000945 1786401103.848201 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x31c89): Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31c89): Mirror 24: 0xc0312952 (stale) Mirror 25: 0xc0312952 (stale) Mirror 26: 0xc0312952 (stale) CRC-32 checksum value for chunk [0, 0x31c89): Mirror 1: 0xdd8e448e Mirror 27: 0xdd8e448e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31c89: skipped 000946 1786401104.425313 trunc from 0x031c89 to 0x02be1d (0x5e6b bytes) 000948 1786401104.570572 trunc from 0x02be1e to 0x01a652 (0x117cb bytes) 000949 1786401104.597082 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a653): Mirror 24: 0xb3a2b22c (stale) Mirror 25: 0xb3a2b22c (stale) Mirror 26: 0xb3a2b22c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a653): Mirror 24: 0xb3a2b22c (stale) Mirror 25: 0xb3a2b22c (stale) Mirror 26: 0xb3a2b22c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a653: skipped 000950 1786401104.928465 MIRROR_RESYNC 4 mirrors 000951 1786401105.676360 mapwrite 0x0229bb thru 0x024a15 (0x205b bytes) 000953 1786401105.813621 mapread 0x01a34b thru 0x01d999 (0x364f bytes) 000954 1786401105.824666 read 0x017648 thru 0x020993 (0x934c bytes) 000955 1786401105.825521 trunc from 0x024a16 to 0x004e80 (0x1fb95 bytes) 000956 1786401105.850069 mapread 0x0003ae thru 0x004e80 (0x4ad3 bytes) 000957 1786401105.873552 read 0x00210e thru 0x004e80 (0x2d73 bytes) 000959 1786401105.874800 mapread 0x000b02 thru 0x003197 (0x2696 bytes) 000963 1786401105.890682 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x4e81): Mirror 25: 0xfe26cd86 (stale) Mirror 26: 0xfe26cd86 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4e81): Mirror 25: 0xfe26cd86 (stale) Mirror 26: 0xfe26cd86 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4e81: skipped 000967 1786401106.283478 mapwrite 0x026f9b thru 0x03264d (0xb6b3 bytes) 000969 1786401106.370588 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3264e): Mirror 26: 0xd2058da1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3264e): Mirror 26: 0xd2058da1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3264e: skipped 000970 1786401106.617415 write 0x029abc thru 0x030ec3 (0x7408 bytes) 000971 1786401106.645380 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3264e): Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3264e): Mirror 26: 0xd2058da1 (stale) CRC-32 checksum value for chunk [0, 0x3264e): Mirror 1: 0xf7d6199e Mirror 27: 0xf7d6199e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3264e: skipped 000972 1786401107.416769 MIRROR_RESYNC 3 mirrors 000973 1786401107.993323 mapread 0x0220dd thru 0x026aaa (0x49ce bytes) 000974 1786401108.013743 mapread 0x029caa thru 0x03264d (0x89a4 bytes) 000975 1786401108.029259 MIRROR_RESYNC 3 mirrors 000976 1786401108.334187 mapread 0x01f733 thru 0x01fdab (0x679 bytes) 000977 1786401108.335789 mapwrite 0x000d47 thru 0x002d0a (0x1fc4 bytes) 000978 1786401108.473141 mapread 0x0119d0 thru 0x018b1f (0x7150 bytes) 000980 1786401108.489180 mapread 0x0139de thru 0x01bc3e (0x8261 bytes) 000981 1786401108.500893 mapread 0x0204f9 thru 0x02ddab (0xd8b3 bytes) 000986 1786401108.523769 write 0x02dd9c thru 0x03b60a (0xd86f bytes) 000988 1786401108.527034 trunc from 0x03b60b to 0x031967 (0x9ca3 bytes) 000989 1786401108.559139 MIRROR_RESYNC 3 mirrors 000992 1786401109.258685 trunc from 0x031968 to 0x02f681 (0x22e6 bytes) 000993 1786401109.324337 mapread 0x00699a thru 0x00bb54 (0x51bb bytes) 000994 1786401109.346361 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2f682): Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f682): Mirror 26: 0x86110337 (stale) Mirror 27: 0x86110337 (stale) CRC-32 checksum value for chunk [0, 0x2f682): Mirror 1: 0x86110337 Mirror 28: 0x86110337 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f682: skipped 000995 1786401110.036994 read 0x0250b1 thru 0x02d55b (0x84ab bytes) All operations completed A-OK! PASS 70a (92s) == sanity-flr test 71: check mirror extend parent fid ==== 18:31:55 (1786401115) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 18:31:59 (1786401119) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.220486 s, 9.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133338 s, 7.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b7:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b7:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d4:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 18:32:09 (1786401129) Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre2 Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5483904 .. Extending file size to 8964608 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8290560 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8846240 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9153824 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 56929 56930 56931 56934 56937 done 192.168.203.137@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg337-client.virtnet /mnt/lustre2 (opts:) 192.168.203.137@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg337-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (79s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 18:33:28 (1786401208) Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre2 Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1488160 .. Extending file size to 3231264 .. done Extending file size to 8149504 .. Extending file size to 7329440 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8354592 .. Extending file size to 7399936 .. Extending file size to 7416192 .. Extending file size to 8649376 .. Extending file size to 8057120 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8581792 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8807872 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8971136 .. Waiting 58262 58263 58265 58266 58267 done 192.168.203.137@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg337-client.virtnet /mnt/lustre2 (opts:) 192.168.203.137@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg337-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (77s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 18:34:45 (1786401285) Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.149937 s, 14.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.252882 s, 8.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x7dbb14a3 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.137@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg337-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 18:35:05 (1786401305) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.120571 s, 17.4 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 18:35:13 (1786401313) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d6:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ce:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 18:35:21 (1786401321) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.141459 s, 14.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ea:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2eb:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 18:35:29 (1786401329) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.186989 s, 11.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ec:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ec:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ed:0x0] } PASS 204b (11s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 18:35:39 (1786401339) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.146017 s, 14.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ee:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ee:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ef:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 18:36:12 (1786401372) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg337-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678446 s, 15.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d3:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d3:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d4:0x0] } PASS 204d (59s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 18:37:11 (1786401431) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg337-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.704237 s, 3.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (57s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 18:38:08 (1786401488) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg337-server mds-ost sync done. Creating new pool test_204f oleg337-server: Pool lustre.test_204f created Adding targets to pool oleg337-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg337-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.189244 s, 11.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg337-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg337-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg337-server: Pool lustre.test_204f destroyed PASS 204f (75s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 18:39:23 (1786401563) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 18:39:30 (1786401570) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 18:39:38 (1786401578) oleg337-server: Pool lustre.test_206 created Adding targets to pool oleg337-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg337-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg337-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg337-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg337-server: Pool lustre.test_206 destroyed PASS 206 (23s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 18:40:01 (1786401601) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443794 s, 23.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349589 s, 30.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xb3e51354 Mirror 3: 0xb3e51354 PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 18:40:11 (1786401611) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 18:40:15 (1786401615) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 18:40:19 (1786401619) Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.118976 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1550000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x15c0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x15c0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x15c0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1300000 Read this (4 bytes): BBBB 192.168.203.137@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg337-client.virtnet /mnt/lustre2 (opts:) PASS 209a (35s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 18:40:54 (1786401654) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0940469 s, 87.1 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 18:41:02 (1786401662) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745514 s, 14.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1fb lmm_fid: [0x200000404:0x1fb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1fb lmm_fid: [0x200000404:0x1fb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834427 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-10 18:41:04.000000000 -0400 Modify: 2026-08-10 18:41:04.000000000 -0400 Change: 2026-08-10 18:41:05.000000000 -0400 Birth: 2026-08-10 18:41:04.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 18:41:09 (1786401669) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 18:41:13 (1786401673) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.475236 s, 86.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0826645 s, 49.5 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 10 18:41 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1fd:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 10 18:41 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 18:41:22 (1786401682) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 18:41:30 (1786401690) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1704 sec =========== 18:41:40 (1786401700) === sanity-flr: start cleanup 18:41:41 (1786401701) === === sanity-flr: finish cleanup 18:41:47 (1786401707) ===