-----============= acceptance-small: sanity-flr ============----- Mon Sep 7 22:12:37 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg651-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 22:12:55 (1788833575) === oleg651-client.virtnet: executing check_config_client /mnt/lustre oleg651-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg651-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9ae489eb2000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9ae489eb2000.idle_timeout=debug disable quota as required oleg651-server: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 22:13:26 (1788833606) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7636] == sanity-flr test 0a: lfs mirror create with -N option == 22:13:28 (1788833608) PASS 0a (16s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 22:13:44 (1788833624) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 22:13:48 (1788833628) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 22:13:50 (1788833630) PASS 0d (22s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 22:14:12 (1788833652) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 22:14:15 (1788833655) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 22:14:18 (1788833658) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9ae489eb2000.stats=clear osc.lustre-OST0001-osc-ffff9ae489eb2000.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 22:14:26 (1788833666) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 22:14:37 (1788833677) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 22:14:44 (1788833684) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 22:14:47 (1788833687) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 22:14:54 (1788833694) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 22:15:02 (1788833702) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 22:15:09 (1788833709) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 22:15:13 (1788833713) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 22:15:20 (1788833720) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 22:15:30 (1788833730) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 22:15:36 (1788833736) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 2.0739 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109977 s, 9.5 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 22:15:47 (1788833747) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.127849 s, 32.8 MB/s PASS 22 (7s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 22:15:55 (1788833755) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 22:16:05 (1788833765) Stopping /mnt/lustre-ost1 (opts:) on oleg651-server oleg651-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ae489eb2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ae489eb2000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg651-server: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg651-client: oleg651-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg651-server: oleg651-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg651-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg651-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg651-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg651-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg651-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg651-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg651-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg651-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg651-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg651-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg651-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg651-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg651-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg651-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg651-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg651-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg651-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg651-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg651-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg651-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg651-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg651-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg651-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg651-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg651-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg651-server oleg651-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9ae489eb2000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ae489eb2000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg651-server: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg651-client: oleg651-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg651-server: oleg651-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg651-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (125s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 22:18:11 (1788833891) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.123918 s, 16.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg651-server oleg651-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ae489eb2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ae489eb2000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg651-server: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg651-client: oleg651-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg651-server: oleg651-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg651-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (35s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 22:18:46 (1788833926) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg651-server oleg651-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ae489eb2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ae489eb2000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg651-server: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg651-client: oleg651-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg651-server: oleg651-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg651-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg651-server oleg651-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9ae489eb2000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ae489eb2000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg651-server: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg651-client: oleg651-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg651-server: oleg651-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg651-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (64s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 22:19:50 (1788833990) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.03795 s, 13.0 MB/s -rw-r--r-- 1 root root 52428800 Sep 7 22:19 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.880s user 0m0.000s sys 0m0.165s Stopping /mnt/lustre-ost1 (opts:) on oleg651-server oleg651-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ae489eb2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ae489eb2000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.885s user 0m0.000s sys 0m0.166s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg651-server: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg651-client: oleg651-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg651-server: oleg651-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg651-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg651-server oleg651-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9ae489eb2000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ae489eb2000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.127s user 0m0.000s sys 0m0.176s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg651-server: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg651-client: oleg651-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg651-server: oleg651-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg651-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg651-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9ae489eb2000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ae489eb2000.ost_server_uuid in FULL state after 0 sec PASS 33b (74s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 22:21:04 (1788834064) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 22:21:07 (1788834067) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 22:21:10 (1788834070) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 22:21:13 (1788834073) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 22:21:20 (1788834080) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.541066 s, 23.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 22:21:31 (1788834091) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317194 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372503 s, 28.1 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 22:21:37 (1788834097) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 22:21:45 (1788834105) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00106391 s, 962 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000684137 s, 1.5 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00237074 s, 432 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00075109 s, 1.4 MB/s flv=10 olv=10 PASS 36d (8s) == sanity-flr test 37: mirror I/O API verification ======= 22:21:53 (1788834113) create test files with size 15738977 .. 5a414dd5505750a4d3ec3b2baf354349 - eaf046b78353a220a159b61f52523076 - 2f6e1edfbcc0b8666d57f0e732004427 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg651-client.virtnet /mnt/lustre (opts:) Starting client: oleg651-client.virtnet: -o user_xattr,flock 192.168.206.151@tcp:/lustre /mnt/lustre PASS 37 (21s) == sanity-flr test 38: resync ============================ 22:22:15 (1788834135) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (12s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 22:22:27 (1788834147) PASS 39 (5s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 22:22:33 (1788834153) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107117 s, 9.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcb:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcc:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029604 s, 35.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd1:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd2:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 22:22:40 (1788834160) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.163101 s, 12.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.383295 s, 10.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.245869 s, 8.5 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 22:22:57 (1788834177) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 22:23:00 (1788834180) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 22:23:03 (1788834183) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245275 s, 42.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 22:23:11 (1788834191) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 22:23:14 (1788834194) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd9:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd4:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd4:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 22:23:23 (1788834203) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.290254 s, 36.1 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg651-client.virtnet /mnt/lustre (opts:) Starting client: oleg651-client.virtnet: -o user_xattr,flock 192.168.206.151@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 22:23:33 (1788834213) llite.lustre-ffff9ae48a268000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.234461 s, 44.7 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg651-client.virtnet /mnt/lustre (opts:) Starting client: oleg651-client.virtnet: -o user_xattr,flock 192.168.206.151@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9ae48482b800.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 22:23:46 (1788834226) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (35s) == sanity-flr test 46: Verify setstripe --copy option ==== 22:24:21 (1788834261) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (34s) == sanity-flr test 47: Verify mirror obj alloc =========== 22:24:55 (1788834295) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 22:24:58 (1788834298) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.205673 s, 15.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.140016 s, 22.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x4631f668 (nosync,stale) Mirror 2: 0x4631f668 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 84238aa684db971dd465fddfd5a0f5f8 - mirror 1 checksum: 86890cc1e32f2c0abb6fd3d8d009158a - mirror 2 checksum: 84238aa684db971dd465fddfd5a0f5f8 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-07T22:25:02-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe9:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-07T22:25:02-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xea:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 84238aa684db971dd465fddfd5a0f5f8 - mirror 1 checksum: 86890cc1e32f2c0abb6fd3d8d009158a - mirror 2 checksum: 86890cc1e32f2c0abb6fd3d8d009158a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-07T22:25:02-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe9:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-07T22:25:02-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xea:0x0] } PASS 48 (10s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 22:25:10 (1788834310) Failing mds1 on oleg651-server Stopping /mnt/lustre-mds1 (opts:) on oleg651-server 22:25:16 (1788834316) shut down facet: mds1 facet_host: oleg651-server facet_failover_host: oleg651-server Failover mds1 to oleg651-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg651-server: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg651-client: oleg651-server: ssh exited with exit code 1 Started lustre-MDT0000 22:25:40 (1788834340) targets are mounted 22:25:40 (1788834340) facet_failover done oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256353 s, 40.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xeb:0x0] } PASS 50A (46s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 22:25:56 (1788834356) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 22:25:59 (1788834359) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 22:26:03 (1788834363) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0157277 s, 1.0 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 22:26:10 (1788834370) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 22:26:14 (1788834374) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00842894 s, 486 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0706606 s, 58.0 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 22:26:21 (1788834381) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 22:26:45 (1788834405) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 22:27:13 (1788834433) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (22s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 22:27:35 (1788834455) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 22:28:00 (1788834480) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 22:28:07 (1788834487) mirror create pid 54323 mirror split pid 54324 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54323 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54324 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 22:29:15 (1788834555) Seed set to 4557 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1788834557.291040 write 0x00d56c thru 0x017fbd (0xaa52 bytes) 000002 1788834557.306357 read 0x001c09 thru 0x00f585 (0xd97d bytes) 000003 1788834557.319160 read 0x0082b1 thru 0x00a68f (0x23df bytes) truncating to largest ever: 0x3c75b 000005 1788834557.321338 trunc from 0x017fbe to 0x03c75a (0x2479d bytes) 000006 1788834557.347841 write 0x03c687 thru 0x03c8a7 (0x221 bytes) 000007 1788834557.386485 mapwrite 0x0344a0 thru 0x038b17 (0x4678 bytes) 000008 1788834557.529349 mapread 0x02bb4d thru 0x031b99 (0x604d bytes) 000009 1788834557.534964 mapwrite 0x026056 thru 0x02a3b0 (0x435b bytes) 000010 1788834557.607799 trunc from 0x03c8a8 to 0x026c9f (0x15c08 bytes) 000011 1788834557.625956 trunc from 0x026ca0 to 0x023856 (0x3449 bytes) 000012 1788834557.647477 trunc from 0x023857 to 0x03afa3 (0x1774d bytes) 000014 1788834557.693663 mapwrite 0x00e950 thru 0x01668d (0x7d3e bytes) 000015 1788834557.796614 mapwrite 0x01a85a thru 0x01eb7c (0x4323 bytes) 000016 1788834557.862927 write 0x016e75 thru 0x0260f4 (0xf280 bytes) 000017 1788834557.870964 mapwrite 0x03d51e thru 0x03ffff (0x2ae2 bytes) 000018 1788834557.964519 read 0x03ba15 thru 0x03d8db (0x1ec7 bytes) 000019 1788834557.993201 mapread 0x0327f1 thru 0x03ffff (0xd80f bytes) 000022 1788834557.997460 mapwrite 0x00e905 thru 0x017eed (0x95e9 bytes) 000023 1788834558.085270 MIRROR_EXTEND to 2 mirrors 000024 1788834558.923547 mapread 0x00e39f thru 0x01256c (0x41ce bytes) 000027 1788834558.988357 read 0x015984 thru 0x0159c8 (0x45 bytes) 000028 1788834558.998432 mapread 0x010544 thru 0x013c02 (0x36bf bytes) 000029 1788834559.017479 mapwrite 0x00bc49 thru 0x00d19e (0x1556 bytes) 000030 1788834559.243927 trunc from 0x040000 to 0x01184a (0x2e7b5 bytes) 000033 1788834559.297543 mapread 0x0067df thru 0x006fbd (0x7df bytes) 000034 1788834559.299650 write 0x0382b8 thru 0x03ffff (0x7d48 bytes) 000035 1788834559.302407 punch from 0x027d53 to 0x034b78 (0xce26 bytes) 000036 1788834559.321362 read 0x006a7d thru 0x010f75 (0xa4f9 bytes) 000037 1788834559.328824 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) 000038 1788834559.548160 write 0x01379d thru 0x01972c (0x5f90 bytes) 000039 1788834559.564046 write 0x010e42 thru 0x019c6d (0x8e2c bytes) 000040 1788834559.578093 mapwrite 0x027e69 thru 0x0364ec (0xe684 bytes) 000041 1788834559.694950 read 0x010a68 thru 0x013885 (0x2e1e bytes) 000042 1788834559.695528 write 0x00d7bc thru 0x0197eb (0xc030 bytes) 000044 1788834559.702758 mapread 0x0260d3 thru 0x02f871 (0x979f bytes) 000049 1788834559.706984 read 0x01e8b5 thru 0x0230ad (0x47f9 bytes) 000051 1788834559.718961 punch from 0x0095f3 to 0x010606 (0x7014 bytes) 000052 1788834559.732837 read 0x00f6aa thru 0x018c1c (0x9573 bytes) 000053 1788834559.744510 MIRROR_EXTEND to 2 mirrors 000054 1788834560.616717 mapwrite 0x034f3b thru 0x03f7a3 (0xa869 bytes) 000056 1788834560.850300 punch from 0x03e537 to 0x03ffff (0x1ac9 bytes) 000057 1788834560.859924 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) 000063 1788834561.008977 MIRROR_EXTEND to 2 mirrors 000066 1788834562.077977 trunc from 0x040000 to 0x005c4e (0x3a3b1 bytes) 000067 1788834562.197505 mapwrite 0x012aae thru 0x02146e (0xe9c1 bytes) 000068 1788834562.411864 trunc from 0x02146f to 0x022e62 (0x19f4 bytes) 000069 1788834562.442658 punch from 0x00cc22 to 0x00ed9b (0x217a bytes) 000073 1788834562.464347 mapwrite 0x0114ed thru 0x01ff43 (0xea57 bytes) 000074 1788834562.543137 punch from 0x00681d to 0x008107 (0x18eb bytes) 000076 1788834562.553204 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x22e63): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22e63): Mirror 2: 0x4b81bc34 (stale) CRC-32 checksum value for chunk [0, 0x22e63): Mirror 1: 0xc5f5fed Mirror 3: 0xc5f5fed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22e63: skipped 000079 1788834563.662286 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) 000081 1788834564.115620 write 0x01b77a thru 0x02b4d1 (0xfd58 bytes) 000082 1788834564.206847 write 0x016827 thru 0x018a28 (0x2202 bytes) 000083 1788834564.218403 mapread 0x00f006 thru 0x010495 (0x1490 bytes) 000084 1788834564.231484 write 0x02ac41 thru 0x031be7 (0x6fa7 bytes) 000085 1788834564.233315 write 0x026fb9 thru 0x027056 (0x9e bytes) 000086 1788834564.235394 punch from 0x00edeb to 0x0118f7 (0x2b0d bytes) 000087 1788834564.263550 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x31be8): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31be8): Mirror 3: 0xfcb99156 (stale) CRC-32 checksum value for chunk [0, 0x31be8): Mirror 1: 0x89eb576 Mirror 4: 0x89eb576 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31be8: skipped 000089 1788834565.256117 MIRROR_RESYNC 3 mirrors 000091 1788834566.412290 write 0x025fcb thru 0x028c83 (0x2cb9 bytes) 000092 1788834566.494289 mapread 0x008a1b thru 0x012dca (0xa3b0 bytes) 000094 1788834566.517796 mapwrite 0x02d400 thru 0x039a7b (0xc67c bytes) 000095 1788834566.694935 read 0x0171df thru 0x0219fd (0xa81f bytes) 000096 1788834566.703733 write 0x009380 thru 0x01525d (0xbede bytes) 000097 1788834566.710226 punch from 0x02e41b to 0x039a7b (0xb661 bytes) 000100 1788834566.723417 read 0x017d49 thru 0x01d0ec (0x53a4 bytes) 000101 1788834566.724278 mapwrite 0x017aa8 thru 0x018526 (0xa7f bytes) 000102 1788834566.793681 punch from 0x02b589 to 0x034282 (0x8cfa bytes) 000103 1788834566.808838 punch from 0x024819 to 0x02cfe6 (0x87ce bytes) 000104 1788834566.814320 mapread 0x013ea3 thru 0x021a8f (0xdbed bytes) 000105 1788834566.833527 mapread 0x039666 thru 0x039a7b (0x416 bytes) 000107 1788834566.834954 trunc from 0x039a7c to 0x03b6fb (0x1c80 bytes) 000108 1788834566.853260 write 0x01bdf8 thru 0x027096 (0xb29f bytes) 000110 1788834566.855749 trunc from 0x03b6fc to 0x0252ba (0x16441 bytes) 000111 1788834566.878912 mapwrite 0x00b2f1 thru 0x01a3f2 (0xf102 bytes) 000114 1788834566.968496 trunc from 0x0252bb to 0x01345a (0x11e60 bytes) 000116 1788834567.010296 write 0x022a75 thru 0x024ecc (0x2458 bytes) 000117 1788834567.013792 write 0x014f24 thru 0x017764 (0x2841 bytes) 000118 1788834567.054649 mapwrite 0x02267e thru 0x02db01 (0xb484 bytes) 000120 1788834567.227770 trunc from 0x02db02 to 0x008cd8 (0x24e29 bytes) 000122 1788834567.258365 write 0x017113 thru 0x01c2f4 (0x51e2 bytes) 000126 1788834567.268596 mapwrite 0x0175b9 thru 0x0207de (0x9226 bytes) 000127 1788834567.465485 mapwrite 0x034153 thru 0x03d7bb (0x9669 bytes) 000128 1788834567.578673 mapwrite 0x012e7b thru 0x0146ea (0x1870 bytes) 000129 1788834567.646381 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d7bc): Mirror 4: 0xbd90dc89 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d7bc): Mirror 4: 0xbd90dc89 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d7bc: skipped 000130 1788834568.046657 mapread 0x00ab5c thru 0x0161d0 (0xb675 bytes) 000131 1788834568.079517 mapread 0x019a50 thru 0x0279f3 (0xdfa4 bytes) 000132 1788834568.104679 write 0x02eb14 thru 0x034538 (0x5a25 bytes) 000133 1788834568.124292 punch from 0x027bb3 to 0x0332f6 (0xb744 bytes) 000136 1788834568.163955 mapread 0x032098 thru 0x035303 (0x326c bytes) 000137 1788834568.181456 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) 000140 1788834568.411974 write 0x025cc5 thru 0x02934a (0x3686 bytes) 000141 1788834568.434567 punch from 0x00e58f to 0x011cf2 (0x3764 bytes) 000144 1788834568.471139 MIRROR_EXTEND to 2 mirrors 000145 1788834569.227620 read 0x01541b thru 0x020ef9 (0xbadf bytes) 000146 1788834569.250399 trunc from 0x03d7bc to 0x00379f (0x3a01c bytes) 000147 1788834569.376856 MIRROR_RESYNC 2 mirrors 000148 1788834570.085056 punch from 0x000c19 to 0x00379f (0x2b87 bytes) 000150 1788834570.191504 mapwrite 0x024091 thru 0x02a01b (0x5f8b bytes) 000151 1788834570.294612 mapread 0x0201b4 thru 0x02038c (0x1d9 bytes) 000152 1788834570.295651 read 0x01131b thru 0x013313 (0x1ff9 bytes) 000153 1788834570.301219 trunc from 0x02a01c to 0x017f6e (0x120ad bytes) 000157 1788834570.334476 mapread 0x0067c5 thru 0x0095ee (0x2e2a bytes) 000160 1788834570.353131 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) 000161 1788834570.511838 mapwrite 0x006b6a thru 0x011e2a (0xb2c1 bytes) 000162 1788834570.610834 trunc from 0x017f6f to 0x017d4f (0x21f bytes) 000164 1788834570.655907 punch from 0x00ab34 to 0x015233 (0xa700 bytes) 000167 1788834570.669831 trunc from 0x017d50 to 0x024901 (0xcbb2 bytes) 000170 1788834570.692419 punch from 0x022c08 to 0x024901 (0x1cfa bytes) 000172 1788834570.698676 trunc from 0x024902 to 0x020ed5 (0x3a2c bytes) 000174 1788834570.719246 punch from 0x01fc4b to 0x020ed5 (0x128b bytes) 000176 1788834570.747546 trunc from 0x020ed6 to 0x00be3f (0x15096 bytes) 000177 1788834570.767617 trunc from 0x00be40 to 0x03982a (0x2d9eb bytes) 000180 1788834570.791960 MIRROR_EXTEND to 2 mirrors 000181 1788834571.719663 mapread 0x01fc03 thru 0x0229ad (0x2dab bytes) 000182 1788834571.776910 write 0x002273 thru 0x01150f (0xf29d bytes) 000183 1788834571.882442 trunc from 0x03982b to 0x027363 (0x124c7 bytes) 000184 1788834571.912065 read 0x0259bb thru 0x027363 (0x19a9 bytes) 000185 1788834571.919335 mapread 0x01b398 thru 0x027363 (0xbfcc bytes) 000186 1788834571.932347 mapwrite 0x02e42d thru 0x03d7f3 (0xf3c7 bytes) 000187 1788834572.031374 mapwrite 0x02ebea thru 0x033d09 (0x5120 bytes) 000188 1788834572.088066 read 0x01a966 thru 0x01fd38 (0x53d3 bytes) 000189 1788834572.088822 mapwrite 0x03dfa9 thru 0x03ffff (0x2057 bytes) 000190 1788834572.148569 mapread 0x005b9f thru 0x014dbf (0xf221 bytes) 000192 1788834572.151367 write 0x03e024 thru 0x03ffff (0x1fdc bytes) 000194 1788834572.153607 read 0x006d39 thru 0x015e91 (0xf159 bytes) 000196 1788834572.160500 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xad360c48 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xaf2bdccd Mirror 3: 0xaf2bdccd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000198 1788834573.145276 read 0x01e78b thru 0x0222c8 (0x3b3e bytes) 000199 1788834573.160577 read 0x02ef0a thru 0x03da4e (0xeb45 bytes) 000200 1788834573.201924 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xad360c48 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xaf2bdccd Mirror 3: 0xaf2bdccd Mirror 4: 0xaf2bdccd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000201 1788834574.277661 punch from 0x015cae to 0x01c1d9 (0x652c bytes) 000202 1788834574.385100 mapread 0x002e02 thru 0x004f57 (0x2156 bytes) 000203 1788834574.443903 read 0x015ce6 thru 0x01c6f8 (0x6a13 bytes) 000206 1788834574.470132 MIRROR_RESYNC 4 mirrors 000207 1788834575.475596 trunc from 0x040000 to 0x02551a (0x1aae5 bytes) 000209 1788834575.527689 write 0x01fa73 thru 0x02df83 (0xe511 bytes) 000210 1788834575.534248 mapwrite 0x031236 thru 0x03bb44 (0xa90f bytes) 000211 1788834575.605448 punch from 0x0384de to 0x03bb44 (0x3667 bytes) 000212 1788834575.610124 trunc from 0x03bb45 to 0x01a716 (0x2142e bytes) 000213 1788834575.627347 MIRROR_RESYNC 4 mirrors 000214 1788834576.691746 read 0x00e4f0 thru 0x018e15 (0xa926 bytes) 000215 1788834576.702238 trunc from 0x01a717 to 0x02940c (0xecf6 bytes) 000217 1788834576.813421 read 0x00b9b2 thru 0x013020 (0x766f bytes) 000218 1788834576.845429 trunc from 0x02940d to 0x035038 (0xbc2c bytes) 000219 1788834576.873593 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x35039): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35039): Mirror 2: 0x9997b225 (stale) Mirror 3: 0x9997b225 (stale) Mirror 4: 0x9997b225 (stale) CRC-32 checksum value for chunk [0, 0x35039): Mirror 1: 0x9997b225 Mirror 5: 0x9997b225 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35039: skipped 000220 1788834578.267571 mapread 0x02efd6 thru 0x035038 (0x6063 bytes) truncating to largest ever: 0x3cea5 000221 1788834578.306826 trunc from 0x035039 to 0x03cea4 (0x7e6c bytes) 000223 1788834578.441166 mapread 0x02580c thru 0x02838d (0x2b82 bytes) 000224 1788834578.453264 trunc from 0x03cea5 to 0x038254 (0x4c50 bytes) 000225 1788834578.480658 mapwrite 0x02c58f thru 0x0351f9 (0x8c6b bytes) 000226 1788834578.574524 write 0x03c670 thru 0x03e105 (0x1a96 bytes) 000227 1788834578.576767 trunc from 0x03e106 to 0x039897 (0x486e bytes) 000228 1788834578.595366 punch from 0x02501f to 0x02f76e (0xa750 bytes) 000229 1788834578.601044 read 0x02fee8 thru 0x039897 (0x99b0 bytes) 000230 1788834578.616481 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x39898): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39898): Mirror 2: 0x4ed528 (stale) Mirror 3: 0x4ed528 (stale) Mirror 4: 0x4ed528 (stale) Mirror 5: 0x4ed528 (stale) CRC-32 checksum value for chunk [0, 0x39898): Mirror 1: 0xea4cac0e Mirror 6: 0xea4cac0e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39898: skipped 000233 1788834580.155299 mapwrite 0x02bba3 thru 0x035c5a (0xa0b8 bytes) 000234 1788834580.492805 mapread 0x001346 thru 0x004b73 (0x382e bytes) 000235 1788834580.512395 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x39898): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39898): Mirror 2: 0x4ed528 (stale) Mirror 3: 0x4ed528 (stale) Mirror 4: 0x4ed528 (stale) Mirror 5: 0x4ed528 (stale) Mirror 6: 0xea4cac0e (stale) CRC-32 checksum value for chunk [0, 0x39898): Mirror 1: 0xad6f0c42 Mirror 7: 0xad6f0c42 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39898: skipped 000236 1788834581.405366 trunc from 0x039898 to 0x00584f (0x34048 bytes) 000238 1788834581.591188 punch from 0x004d4d to 0x00584f (0xb03 bytes) 000239 1788834581.599053 write 0x02627a thru 0x03430d (0xe094 bytes) 000240 1788834581.604930 read 0x019b50 thru 0x01ce7f (0x3330 bytes) 000241 1788834581.612221 write 0x01b8dc thru 0x01e107 (0x282c bytes) 000242 1788834581.618430 mapread 0x01f3ca thru 0x023a0e (0x4645 bytes) 000243 1788834581.664538 MIRROR_RESYNC 7 mirrors 000244 1788834583.568592 trunc from 0x03430e to 0x02f97e (0x498f bytes) 000245 1788834583.656125 read 0x017562 thru 0x01bde7 (0x4886 bytes) 000248 1788834583.671641 mapwrite 0x038d32 thru 0x03c79b (0x3a6a bytes) 000250 1788834583.794111 MIRROR_RESYNC 7 mirrors 000251 1788834585.651532 mapread 0x020c46 thru 0x02f53b (0xe8f6 bytes) 000252 1788834585.698861 mapread 0x0338ae thru 0x03c79b (0x8eee bytes) 000253 1788834585.716445 mapread 0x01a1a5 thru 0x029d16 (0xfb72 bytes) 000255 1788834585.733166 write 0x03b1ce thru 0x03ffff (0x4e32 bytes) 000256 1788834585.851536 write 0x02776b thru 0x02d9b3 (0x6249 bytes) 000259 1788834585.868351 mapwrite 0x0397ed thru 0x03ffff (0x6813 bytes) 000260 1788834585.958638 mapread 0x003d88 thru 0x0044e4 (0x75d bytes) 000261 1788834585.983213 write 0x0050e5 thru 0x010cdd (0xbbf9 bytes) 000262 1788834585.986588 punch from 0x03043e to 0x03fb1e (0xf6e1 bytes) 000263 1788834586.013075 punch from 0x007d14 to 0x0092fe (0x15eb bytes) 000266 1788834586.031169 mapread 0x01dca0 thru 0x025632 (0x7993 bytes) 000270 1788834586.059557 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x25842fad (stale) Mirror 4: 0x25842fad (stale) Mirror 5: 0x25842fad (stale) Mirror 6: 0x25842fad (stale) Mirror 7: 0x25842fad (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x25842fad (stale) Mirror 4: 0x25842fad (stale) Mirror 5: 0x25842fad (stale) Mirror 6: 0x25842fad (stale) Mirror 7: 0x25842fad (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000271 1788834586.600635 punch from 0x037108 to 0x03ca37 (0x5930 bytes) 000272 1788834586.607187 mapwrite 0x017cac thru 0x025ae7 (0xde3c bytes) 000274 1788834586.699480 read 0x020db5 thru 0x02270b (0x1957 bytes) 000275 1788834586.700181 read 0x018b1d thru 0x026dc3 (0xe2a7 bytes) 000276 1788834586.701409 mapread 0x01b279 thru 0x02b231 (0xffb9 bytes) 000277 1788834586.716522 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x25842fad (stale) Mirror 4: 0x25842fad (stale) Mirror 5: 0x25842fad (stale) Mirror 6: 0x25842fad (stale) Mirror 7: 0x25842fad (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbe75c40f Mirror 8: 0xbe75c40f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000278 1788834587.946193 mapwrite 0x025cd2 thru 0x035348 (0xf677 bytes) 000280 1788834588.204644 mapwrite 0x037c8b thru 0x03ffff (0x8375 bytes) 000281 1788834588.273367 read 0x03e8ef thru 0x03ffff (0x1711 bytes) 000282 1788834588.275361 write 0x015d83 thru 0x01e1e4 (0x8462 bytes) 000283 1788834588.306829 MIRROR_RESYNC 7 mirrors 000285 1788834590.097812 trunc from 0x040000 to 0x021038 (0x1efc7 bytes) 000286 1788834590.220097 mapread 0x0147de thru 0x01b55a (0x6d7d bytes) 000287 1788834590.255860 mapwrite 0x01d9b2 thru 0x02768d (0x9cdc bytes) 000288 1788834590.346367 trunc from 0x02768e to 0x001393 (0x262fa bytes) 000291 1788834590.363500 write 0x021126 thru 0x02ab1e (0x99f9 bytes) 000293 1788834590.367384 punch from 0x01bb54 to 0x023701 (0x7bae bytes) 000294 1788834590.381969 mapread 0x0014f3 thru 0x002607 (0x1115 bytes) 000295 1788834590.391038 trunc from 0x02ab1f to 0x00a6ec (0x20432 bytes) 000297 1788834590.406591 punch from 0x00310f to 0x00a6ec (0x75de bytes) 000298 1788834590.412364 mapwrite 0x03d103 thru 0x03ffff (0x2efd bytes) 000300 1788834590.506472 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xed9d54de (stale) Mirror 4: 0xed9d54de (stale) Mirror 5: 0xed9d54de (stale) Mirror 6: 0xed9d54de (stale) Mirror 8: 0xed9d54de (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xed9d54de (stale) Mirror 4: 0xed9d54de (stale) Mirror 5: 0xed9d54de (stale) Mirror 6: 0xed9d54de (stale) Mirror 8: 0xed9d54de (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000301 1788834591.000009 read 0x00aeef thru 0x00e29f (0x33b1 bytes) 000302 1788834591.024547 write 0x0382a4 thru 0x03ffff (0x7d5c bytes) 000303 1788834591.042143 punch from 0x03478a to 0x03530b (0xb82 bytes) 000304 1788834591.081072 write 0x036685 thru 0x03ffff (0x997b bytes) 000305 1788834591.099465 mapwrite 0x037002 thru 0x03ffff (0x8ffe bytes) 000306 1788834591.190276 trunc from 0x040000 to 0x01287f (0x2d780 bytes) 000307 1788834591.214530 punch from 0x003ef2 to 0x010558 (0xc667 bytes) 000308 1788834591.230800 write 0x03f6cd thru 0x03ffff (0x933 bytes) 000309 1788834591.233020 punch from 0x00477b to 0x007291 (0x2b17 bytes) 000310 1788834591.244759 mapwrite 0x01a9a8 thru 0x02256f (0x7bc8 bytes) 000311 1788834591.314296 mapread 0x01f05d thru 0x02e7c8 (0xf76c bytes) 000312 1788834591.333143 read 0x0047c7 thru 0x011364 (0xcb9e bytes) 000313 1788834591.343738 write 0x004d24 thru 0x013f69 (0xf246 bytes) 000315 1788834591.347203 mapread 0x0160c7 thru 0x01ecc1 (0x8bfb bytes) 000316 1788834591.348618 trunc from 0x040000 to 0x02573e (0x1a8c1 bytes) 000318 1788834591.363789 mapread 0x00b7ac thru 0x019548 (0xdd9d bytes) 000319 1788834591.364641 read 0x024cec thru 0x02573e (0xa53 bytes) 000320 1788834591.365135 mapread 0x024b03 thru 0x02573e (0xc3c bytes) 000321 1788834591.370773 mapwrite 0x029605 thru 0x0298f2 (0x2ee bytes) 000322 1788834591.447547 write 0x02ed4a thru 0x033871 (0x4b28 bytes) 000323 1788834591.449468 read 0x02491d thru 0x02ab1e (0x6202 bytes) 000324 1788834591.460150 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33872): Mirror 4: 0xec66de2f (stale) Mirror 5: 0xec66de2f (stale) Mirror 6: 0xec66de2f (stale) Mirror 8: 0xec66de2f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33872): Mirror 4: 0xec66de2f (stale) Mirror 5: 0xec66de2f (stale) Mirror 6: 0xec66de2f (stale) Mirror 8: 0xec66de2f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33872: skipped 000326 1788834591.979919 write 0x00ca49 thru 0x015d49 (0x9301 bytes) 000329 1788834592.003207 mapread 0x0017bf thru 0x004082 (0x28c4 bytes) 000330 1788834592.035206 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x33872): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33872): Mirror 4: 0xec66de2f (stale) Mirror 5: 0xec66de2f (stale) Mirror 6: 0xec66de2f (stale) Mirror 8: 0xec66de2f (stale) CRC-32 checksum value for chunk [0, 0x33872): Mirror 1: 0xd9b3c10f Mirror 9: 0xd9b3c10f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33872: skipped 000332 1788834592.977444 punch from 0x01b9df to 0x02119f (0x57c1 bytes) 000333 1788834593.077212 mapread 0x0209fc thru 0x02d1e1 (0xc7e6 bytes) 000334 1788834593.110283 mapwrite 0x02440b thru 0x02914d (0x4d43 bytes) 000336 1788834593.252328 read 0x01aa7b thru 0x01f56e (0x4af4 bytes) 000337 1788834593.262147 punch from 0x00b17f to 0x00f83a (0x46bc bytes) 000338 1788834593.269681 write 0x02ff0f thru 0x038db7 (0x8ea9 bytes) 000339 1788834593.277148 read 0x017c8c thru 0x01ec07 (0x6f7c bytes) 000340 1788834593.294657 read 0x000738 thru 0x000cee (0x5b7 bytes) 000342 1788834593.295900 trunc from 0x038db8 to 0x021160 (0x17c57 bytes) 000343 1788834593.356445 read 0x01fea6 thru 0x021160 (0x12bb bytes) 000345 1788834593.363904 mapread 0x000a49 thru 0x00ee4d (0xe405 bytes) 000346 1788834593.366801 trunc from 0x021161 to 0x038a8d (0x1792d bytes) 000347 1788834593.385918 punch from 0x01bab1 to 0x01cd8c (0x12dc bytes) 000348 1788834593.391635 mapwrite 0x008528 thru 0x010424 (0x7efd bytes) 000349 1788834593.440686 read 0x002794 thru 0x00c4ca (0x9d37 bytes) 000350 1788834593.444425 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38a8e): Mirror 4: 0xa6d7751a (stale) Mirror 5: 0xa6d7751a (stale) Mirror 6: 0xa6d7751a (stale) Mirror 8: 0xa6d7751a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38a8e): Mirror 4: 0xa6d7751a (stale) Mirror 5: 0xa6d7751a (stale) Mirror 6: 0xa6d7751a (stale) Mirror 8: 0xa6d7751a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38a8e: skipped 000353 1788834593.789905 mapwrite 0x020437 thru 0x02b057 (0xac21 bytes) 000354 1788834593.871964 trunc from 0x038a8e to 0x013b1d (0x24f70 bytes) 000356 1788834593.891625 punch from 0x008c5f to 0x013b1d (0xaebf bytes) 000357 1788834593.898533 trunc from 0x013b1e to 0x008f16 (0xac07 bytes) 000358 1788834593.923433 trunc from 0x008f17 to 0x016358 (0xd442 bytes) 000360 1788834593.944136 punch from 0x0040e4 to 0x00bce0 (0x7bfd bytes) 000364 1788834593.955798 write 0x03aca2 thru 0x03ffff (0x535e bytes) 000365 1788834593.957567 trunc from 0x040000 to 0x031bad (0xe452 bytes) 000366 1788834593.979145 mapwrite 0x0266bc thru 0x031fe9 (0xb92e bytes) 000368 1788834594.080126 MIRROR_RESYNC 5 mirrors 000369 1788834595.834362 trunc from 0x031fea to 0x033c13 (0x1c2a bytes) 000370 1788834595.930811 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x33c14): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33c14): Mirror 4: 0x7fdd4c72 (stale) Mirror 5: 0x7fdd4c72 (stale) Mirror 6: 0x7fdd4c72 (stale) Mirror 8: 0x7fdd4c72 (stale) CRC-32 checksum value for chunk [0, 0x33c14): Mirror 1: 0x7fdd4c72 Mirror 9: 0x7fdd4c72 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33c14: skipped 000371 1788834597.041846 trunc from 0x033c14 to 0x039da4 (0x6191 bytes) 000372 1788834597.189116 MIRROR_RESYNC 6 mirrors 000373 1788834598.926605 mapwrite 0x03b60d thru 0x03e3a4 (0x2d98 bytes) 000374 1788834599.088597 mapread 0x03e054 thru 0x03e3a4 (0x351 bytes) 000375 1788834599.089842 trunc from 0x03e3a5 to 0x025e7e (0x18526 bytes) 000377 1788834599.113154 write 0x0396f2 thru 0x03ffff (0x690e bytes) 000379 1788834599.120057 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x75a6fb6f (stale) Mirror 5: 0x75a6fb6f (stale) Mirror 6: 0x75a6fb6f (stale) Mirror 8: 0x75a6fb6f (stale) Mirror 9: 0x75a6fb6f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7b5a2a88 Mirror 10: 0x7b5a2a88 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000381 1788834600.409987 punch from 0x00a59d to 0x015ef4 (0xb958 bytes) 000382 1788834600.595137 mapread 0x002147 thru 0x00be98 (0x9d52 bytes) 000384 1788834600.618752 mapwrite 0x0379d4 thru 0x03ffff (0x862c bytes) 000386 1788834600.716612 mapwrite 0x03f774 thru 0x03ffff (0x88c bytes) 000387 1788834600.757649 read 0x0353a7 thru 0x03ffff (0xac59 bytes) 000389 1788834600.759107 mapread 0x016fa5 thru 0x0250a0 (0xe0fc bytes) 000392 1788834600.793504 punch from 0x01b42a to 0x027002 (0xbbd9 bytes) 000393 1788834600.800546 mapread 0x00b854 thru 0x00cc64 (0x1411 bytes) 000394 1788834600.804156 punch from 0x023bfe to 0x02bc2e (0x8031 bytes) 000395 1788834600.821435 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x75a6fb6f (stale) Mirror 6: 0x75a6fb6f (stale) Mirror 8: 0x75a6fb6f (stale) Mirror 9: 0x75a6fb6f (stale) Mirror 10: 0x7b5a2a88 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x75a6fb6f (stale) Mirror 6: 0x75a6fb6f (stale) Mirror 8: 0x75a6fb6f (stale) Mirror 9: 0x75a6fb6f (stale) Mirror 10: 0x7b5a2a88 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000397 1788834601.280036 read 0x013ddf thru 0x0213ba (0xd5dc bytes) 000398 1788834601.298967 punch from 0x01cfa7 to 0x027c0b (0xac65 bytes) 000399 1788834601.307574 read 0x03084c thru 0x03b186 (0xa93b bytes) 000400 1788834601.330448 mapread 0x03c49e thru 0x03ffff (0x3b62 bytes) 000401 1788834601.336532 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x75a6fb6f (stale) Mirror 6: 0x75a6fb6f (stale) Mirror 8: 0x75a6fb6f (stale) Mirror 9: 0x75a6fb6f (stale) Mirror 10: 0x7b5a2a88 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe199f81e Mirror 11: 0xe199f81e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000406 1788834602.617459 punch from 0x00a24e to 0x00e3d7 (0x418a bytes) 000408 1788834602.696430 write 0x0036e0 thru 0x00a79c (0x70bd bytes) 000409 1788834602.707756 punch from 0x036f52 to 0x03bfdb (0x508a bytes) 000410 1788834602.724400 write 0x0384bf thru 0x03bcdc (0x381e bytes) 000411 1788834602.735570 mapread 0x028f4c thru 0x02b870 (0x2925 bytes) 000412 1788834602.761603 mapwrite 0x03bb51 thru 0x03ffff (0x44af bytes) 000413 1788834602.822519 write 0x0082c3 thru 0x010de2 (0x8b20 bytes) 000414 1788834602.833147 punch from 0x011268 to 0x02016c (0xef05 bytes) 000415 1788834602.858737 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x75a6fb6f (stale) Mirror 6: 0x75a6fb6f (stale) Mirror 8: 0x75a6fb6f (stale) Mirror 9: 0x75a6fb6f (stale) Mirror 10: 0x7b5a2a88 (stale) Mirror 11: 0xe199f81e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5cc8b79a Mirror 12: 0x5cc8b79a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000418 1788834604.255256 mapwrite 0x03f05b thru 0x03ffff (0xfa5 bytes) 000419 1788834604.565342 mapread 0x01b46a thru 0x02473f (0x92d6 bytes) 000420 1788834604.587721 write 0x00e1b3 thru 0x012538 (0x4386 bytes) 000422 1788834604.599899 write 0x025410 thru 0x028035 (0x2c26 bytes) 000423 1788834604.601332 mapwrite 0x031858 thru 0x033bb4 (0x235d bytes) 000425 1788834604.686631 trunc from 0x040000 to 0x02a9fb (0x15604 bytes) 000427 1788834604.719644 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a9fc): Mirror 4: 0xc93e0c0e (stale) Mirror 6: 0xc93e0c0e (stale) Mirror 8: 0xc93e0c0e (stale) Mirror 9: 0xc93e0c0e (stale) Mirror 11: 0x648bf143 (stale) Mirror 12: 0xf4aef937 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a9fc): Mirror 4: 0xc93e0c0e (stale) Mirror 6: 0xc93e0c0e (stale) Mirror 8: 0xc93e0c0e (stale) Mirror 9: 0xc93e0c0e (stale) Mirror 11: 0x648bf143 (stale) Mirror 12: 0xf4aef937 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a9fc: skipped 000428 1788834605.331293 MIRROR_RESYNC 7 mirrors 000429 1788834606.850445 punch from 0x0101bf to 0x0145fa (0x443c bytes) 000430 1788834607.043107 mapread 0x01bdba thru 0x023720 (0x7967 bytes) 000431 1788834607.098994 mapwrite 0x006329 thru 0x00815a (0x1e32 bytes) 000432 1788834607.255628 mapwrite 0x028492 thru 0x036fcc (0xeb3b bytes) 000434 1788834607.358683 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36fcd): Mirror 6: 0xdc2dd5f0 (stale) Mirror 8: 0xdc2dd5f0 (stale) Mirror 9: 0xdc2dd5f0 (stale) Mirror 11: 0xdc2dd5f0 (stale) Mirror 12: 0xdc2dd5f0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36fcd): Mirror 6: 0xdc2dd5f0 (stale) Mirror 8: 0xdc2dd5f0 (stale) Mirror 9: 0xdc2dd5f0 (stale) Mirror 11: 0xdc2dd5f0 (stale) Mirror 12: 0xdc2dd5f0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36fcd: skipped 000435 1788834607.730567 read 0x00e4d3 thru 0x00f2b8 (0xde6 bytes) 000438 1788834607.738015 write 0x01fd3f thru 0x025bd6 (0x5e98 bytes) 000439 1788834607.751848 MIRROR_RESYNC 6 mirrors 000440 1788834608.934093 read 0x03248f thru 0x036fcc (0x4b3e bytes) 000443 1788834608.950038 mapread 0x028d48 thru 0x02b20b (0x24c4 bytes) 000444 1788834608.973408 MIRROR_RESYNC 6 mirrors 000446 1788834609.364413 read 0x008f8b thru 0x00fd6b (0x6de1 bytes) 000449 1788834609.374857 read 0x027b93 thru 0x02e7af (0x6c1d bytes) 000452 1788834609.375760 mapwrite 0x01f34a thru 0x02a57b (0xb232 bytes) 000453 1788834609.673857 read 0x000f5a thru 0x005796 (0x483d bytes) 000454 1788834609.723224 mapread 0x03275d thru 0x036fcc (0x4870 bytes) 000457 1788834609.727116 punch from 0x00ac40 to 0x01abe3 (0xffa4 bytes) 000460 1788834609.741681 trunc from 0x036fcd to 0x03cc50 (0x5c84 bytes) 000461 1788834609.786898 write 0x0213bd thru 0x02a64d (0x9291 bytes) 000462 1788834609.788072 punch from 0x00d3a8 to 0x016352 (0x8fab bytes) truncating to largest ever: 0x3d14f 000463 1788834609.810834 trunc from 0x03cc51 to 0x03d14e (0x4fe bytes) 000464 1788834609.825116 read 0x00f34d thru 0x01bb60 (0xc814 bytes) 000465 1788834609.837180 trunc from 0x03d14f to 0x01648c (0x26cc2 bytes) 000466 1788834609.857520 trunc from 0x01648d to 0x03b01f (0x24b93 bytes) 000467 1788834609.874218 trunc from 0x03b020 to 0x0360e2 (0x4f3d bytes) 000468 1788834609.893771 trunc from 0x0360e3 to 0x01824f (0x1de93 bytes) 000469 1788834609.925621 mapwrite 0x033394 thru 0x0358b7 (0x2524 bytes) 000471 1788834610.108621 mapread 0x000520 thru 0x00a258 (0x9d39 bytes) 000472 1788834610.114546 write 0x002432 thru 0x00a63c (0x820b bytes) 000474 1788834610.122954 mapwrite 0x0221b0 thru 0x02b30c (0x915d bytes) 000475 1788834610.355168 punch from 0x01d8e7 to 0x02169b (0x3db5 bytes) 000476 1788834610.360986 mapwrite 0x016462 thru 0x01c075 (0x5c14 bytes) 000478 1788834610.463194 mapread 0x033b89 thru 0x0358b7 (0x1d2f bytes) 000479 1788834610.474317 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x358b8): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x358b8): Mirror 6: 0x1da5995a (stale) Mirror 8: 0x1da5995a (stale) Mirror 9: 0x1da5995a (stale) Mirror 11: 0x1da5995a (stale) Mirror 12: 0x1da5995a (stale) CRC-32 checksum value for chunk [0, 0x358b8): Mirror 1: 0x197a9977 Mirror 13: 0x197a9977 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x358b8: skipped 000480 1788834611.461041 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x358b8): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x358b8): Mirror 6: 0x1da5995a (stale) Mirror 8: 0x1da5995a (stale) Mirror 9: 0x1da5995a (stale) Mirror 11: 0x1da5995a (stale) Mirror 12: 0x1da5995a (stale) CRC-32 checksum value for chunk [0, 0x358b8): Mirror 1: 0x197a9977 Mirror 13: 0x197a9977 Mirror 14: 0x197a9977 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x358b8: skipped 000482 1788834612.645417 mapread 0x0329ba thru 0x0358b7 (0x2efe bytes) 000483 1788834612.666158 trunc from 0x0358b8 to 0x0015be (0x342f9 bytes) 000485 1788834612.887624 write 0x009c85 thru 0x00f6c0 (0x5a3c bytes) 000486 1788834612.889582 trunc from 0x00f6c1 to 0x022896 (0x131d6 bytes) 000487 1788834612.928525 mapread 0x0163d8 thru 0x01e434 (0x805d bytes) 000488 1788834612.950683 punch from 0x00731e to 0x00e634 (0x7317 bytes) 000489 1788834612.967865 trunc from 0x022897 to 0x009bfd (0x18c99 bytes) 000491 1788834612.995633 mapread 0x008a61 thru 0x009bfd (0x119d bytes) 000494 1788834613.013610 trunc from 0x009bfe to 0x02d710 (0x23b13 bytes) 000495 1788834613.029528 read 0x02be13 thru 0x02d710 (0x18fe bytes) 000497 1788834613.035824 punch from 0x00aee6 to 0x01a897 (0xf9b2 bytes) 000498 1788834613.042044 write 0x030ffa thru 0x038b86 (0x7b8d bytes) 000499 1788834613.044427 read 0x00033d thru 0x008175 (0x7e39 bytes) 000500 1788834613.058381 mapread 0x023521 thru 0x027aeb (0x45cb bytes) 000501 1788834613.060775 mapwrite 0x00a0a6 thru 0x01445b (0xa3b6 bytes) 000502 1788834613.237620 mapwrite 0x028d91 thru 0x03125a (0x84ca bytes) 000504 1788834613.377144 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x38b87): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38b87): Mirror 6: 0x8a6b5363 (stale) Mirror 8: 0x8a6b5363 (stale) Mirror 9: 0x8a6b5363 (stale) Mirror 11: 0x8a6b5363 (stale) Mirror 12: 0x8a6b5363 (stale) Mirror 13: 0xbc4e27ba (stale) Mirror 14: 0xbc4e27ba (stale) CRC-32 checksum value for chunk [0, 0x38b87): Mirror 1: 0xf0e50348 Mirror 15: 0xf0e50348 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38b87: skipped 000505 1788834614.509388 trunc from 0x038b87 to 0x01b99c (0x1d1ea bytes) 000506 1788834614.657161 read 0x00289d thru 0x006e66 (0x45ca bytes) truncating to largest ever: 0x3e6ac 000507 1788834614.666456 trunc from 0x01b99d to 0x03e6ab (0x22d0f bytes) 000508 1788834614.687233 read 0x02b9d4 thru 0x02e48c (0x2ab9 bytes) 000509 1788834614.704492 mapwrite 0x02c9f5 thru 0x030b26 (0x4132 bytes) 000510 1788834614.762202 mapread 0x014c18 thru 0x01e3d5 (0x97be bytes) 000511 1788834614.764351 punch from 0x036d42 to 0x03e6ab (0x796a bytes) 000512 1788834614.769507 read 0x008cd1 thru 0x010e1d (0x814d bytes) 000513 1788834614.770243 punch from 0x00643d to 0x009a9b (0x365f bytes) 000514 1788834614.777102 read 0x001534 thru 0x003de3 (0x28b0 bytes) 000515 1788834614.777779 mapwrite 0x01d107 thru 0x02281d (0x5717 bytes) 000516 1788834614.824981 mapread 0x0107e9 thru 0x020576 (0xfd8e bytes) 000517 1788834614.825930 mapwrite 0x035655 thru 0x0386f7 (0x30a3 bytes) 000518 1788834614.928230 punch from 0x00c88a to 0x01b841 (0xefb8 bytes) 000519 1788834614.935406 mapwrite 0x00e902 thru 0x017643 (0x8d42 bytes) 000520 1788834615.078134 punch from 0x036760 to 0x0370e5 (0x986 bytes) 000521 1788834615.090131 MIRROR_RESYNC 9 mirrors 000523 1788834616.815510 write 0x034205 thru 0x03c0da (0x7ed6 bytes) 000524 1788834616.986658 read 0x02b0a8 thru 0x034f6e (0x9ec7 bytes) 000525 1788834617.023348 MIRROR_RESYNC 9 mirrors 000527 1788834618.557645 MIRROR_RESYNC 9 mirrors 000528 1788834619.162301 punch from 0x0232fa to 0x0251c7 (0x1ece bytes) 000529 1788834619.266863 punch from 0x02eb2d to 0x035667 (0x6b3b bytes) 000531 1788834619.286125 write 0x02598d thru 0x030b26 (0xb19a bytes) 000533 1788834619.326537 mapwrite 0x01aad8 thru 0x01cdd1 (0x22fa bytes) 000534 1788834619.450421 mapread 0x029b32 thru 0x030e38 (0x7307 bytes) 000535 1788834619.452186 read 0x017797 thru 0x0188f5 (0x115f bytes) 000536 1788834619.455130 read 0x024a1b thru 0x029d9d (0x5383 bytes) 000537 1788834619.455772 mapread 0x029e5f thru 0x0337ca (0x996c bytes) 000538 1788834619.492347 punch from 0x034994 to 0x03e6ab (0x9d18 bytes) 000540 1788834619.504863 trunc from 0x03e6ac to 0x0193c6 (0x252e5 bytes) 000541 1788834619.538550 punch from 0x00a5d5 to 0x0193c6 (0xedf2 bytes) 000542 1788834619.546624 trunc from 0x0193c7 to 0x012f3b (0x648b bytes) 000543 1788834619.571918 punch from 0x003ef3 to 0x00b8e2 (0x79f0 bytes) 000544 1788834619.593196 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x12f3c): Mirror 6: 0xed8eb6d0 (stale) Mirror 8: 0xed8eb6d0 (stale) Mirror 9: 0xed8eb6d0 (stale) Mirror 11: 0xed8eb6d0 (stale) Mirror 13: 0xed8eb6d0 (stale) Mirror 14: 0xed8eb6d0 (stale) Mirror 15: 0xed8eb6d0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12f3c): Mirror 6: 0xed8eb6d0 (stale) Mirror 8: 0xed8eb6d0 (stale) Mirror 9: 0xed8eb6d0 (stale) Mirror 11: 0xed8eb6d0 (stale) Mirror 13: 0xed8eb6d0 (stale) Mirror 14: 0xed8eb6d0 (stale) Mirror 15: 0xed8eb6d0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12f3c: skipped 000546 1788834620.130357 read 0x000697 thru 0x00e92d (0xe297 bytes) 000548 1788834620.140628 mapwrite 0x01da68 thru 0x01e710 (0xca9 bytes) 000549 1788834620.243442 write 0x033910 thru 0x03d32b (0x9a1c bytes) 000551 1788834620.246456 trunc from 0x03d32c to 0x023710 (0x19c1b bytes) 000553 1788834620.273443 MIRROR_RESYNC 8 mirrors 000554 1788834622.356364 trunc from 0x023711 to 0x02e335 (0xac25 bytes) 000555 1788834622.428895 read 0x026938 thru 0x02e335 (0x79fe bytes) 000556 1788834622.441884 MIRROR_RESYNC 8 mirrors 000559 1788834624.224493 punch from 0x01cbdd to 0x0299c3 (0xcde7 bytes) 000560 1788834624.302200 write 0x021fda thru 0x022014 (0x3b bytes) 000561 1788834624.329802 read 0x021ad1 thru 0x02e335 (0xc865 bytes) 000563 1788834624.343765 read 0x01d320 thru 0x0293b0 (0xc091 bytes) 000564 1788834624.366529 trunc from 0x02e336 to 0x01ddec (0x10549 bytes) 000565 1788834624.392568 write 0x03cf9b thru 0x03ffff (0x3065 bytes) 000566 1788834624.395648 trunc from 0x040000 to 0x0305f7 (0xfa08 bytes) 000568 1788834624.419862 mapread 0x024d3c thru 0x0305f7 (0xb8bc bytes) 000569 1788834624.497643 write 0x03bcdb thru 0x03ffff (0x4325 bytes) 000570 1788834624.499062 trunc from 0x040000 to 0x002f5f (0x3d0a0 bytes) 000571 1788834624.525477 mapread 0x001567 thru 0x002f5f (0x19f9 bytes) 000572 1788834624.528178 trunc from 0x002f60 to 0x027105 (0x241a6 bytes) 000573 1788834624.552173 trunc from 0x027106 to 0x01e0dc (0x9029 bytes) 000575 1788834624.569543 mapwrite 0x0352ab thru 0x03ffff (0xad55 bytes) 000576 1788834624.652540 read 0x02d84e thru 0x03cab9 (0xf26c bytes) 000577 1788834624.666847 punch from 0x012387 to 0x01dbe7 (0xb861 bytes) 000578 1788834624.673262 trunc from 0x040000 to 0x030d23 (0xf2dc bytes) 000579 1788834624.706456 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30d24): Mirror 8: 0xabad6680 (stale) Mirror 9: 0xabad6680 (stale) Mirror 11: 0xabad6680 (stale) Mirror 13: 0xabad6680 (stale) Mirror 14: 0xabad6680 (stale) Mirror 15: 0xabad6680 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30d24): Mirror 8: 0xabad6680 (stale) Mirror 9: 0xabad6680 (stale) Mirror 11: 0xabad6680 (stale) Mirror 13: 0xabad6680 (stale) Mirror 14: 0xabad6680 (stale) Mirror 15: 0xabad6680 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30d24: skipped 000583 1788834625.240552 trunc from 0x030d24 to 0x008aad (0x28276 bytes) 000584 1788834625.260999 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x8aae): Mirror 8: 0x2a6ea9da (stale) Mirror 9: 0x2a6ea9da (stale) Mirror 13: 0x2a6ea9da (stale) Mirror 14: 0x2a6ea9da (stale) Mirror 15: 0x2a6ea9da (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8aae): Mirror 8: 0x2a6ea9da (stale) Mirror 9: 0x2a6ea9da (stale) Mirror 13: 0x2a6ea9da (stale) Mirror 14: 0x2a6ea9da (stale) Mirror 15: 0x2a6ea9da (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8aae: skipped 000585 1788834625.678107 write 0x028d8c thru 0x0366b6 (0xd92b bytes) 000586 1788834625.692223 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x366b7): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x366b7): Mirror 8: 0xd5bb394 (stale) Mirror 9: 0xd5bb394 (stale) Mirror 13: 0xd5bb394 (stale) Mirror 14: 0xd5bb394 (stale) Mirror 15: 0xd5bb394 (stale) CRC-32 checksum value for chunk [0, 0x366b7): Mirror 1: 0xadeb6658 Mirror 16: 0xadeb6658 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x366b7: skipped 000588 1788834627.066230 mapread 0x01fb32 thru 0x0205b2 (0xa81 bytes) 000589 1788834627.089660 mapread 0x034080 thru 0x0366b6 (0x2637 bytes) 000592 1788834627.125043 read 0x02212c thru 0x031b9a (0xfa6f bytes) 000593 1788834627.143104 punch from 0x00c047 to 0x012c97 (0x6c51 bytes) 000594 1788834627.238385 read 0x022491 thru 0x02e21e (0xbd8e bytes) 000595 1788834627.250996 mapwrite 0x025015 thru 0x0297e8 (0x47d4 bytes) 000598 1788834627.306351 read 0x02c378 thru 0x02e510 (0x2199 bytes) 000599 1788834627.307631 mapwrite 0x01a83d thru 0x01d6eb (0x2eaf bytes) 000601 1788834627.357127 write 0x0263ed thru 0x02d43d (0x7051 bytes) 000602 1788834627.359519 punch from 0x00fa8c to 0x01634d (0x68c2 bytes) 000603 1788834627.377659 write 0x01ae0d thru 0x01dd85 (0x2f79 bytes) 000604 1788834627.381997 mapread 0x02a506 thru 0x030d5e (0x6859 bytes) 000605 1788834627.387121 trunc from 0x0366b7 to 0x014040 (0x22676 bytes) 000606 1788834627.430398 mapread 0x00065a thru 0x003af9 (0x34a0 bytes) 000608 1788834627.436186 punch from 0x0056ae to 0x007473 (0x1dc6 bytes) 000610 1788834627.455423 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x14041): Mirror 8: 0x89df76b6 (stale) Mirror 9: 0x89df76b6 (stale) Mirror 13: 0x89df76b6 (stale) Mirror 14: 0x89df76b6 (stale) Mirror 15: 0x89df76b6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14041): Mirror 8: 0x89df76b6 (stale) Mirror 9: 0x89df76b6 (stale) Mirror 13: 0x89df76b6 (stale) Mirror 14: 0x89df76b6 (stale) Mirror 15: 0x89df76b6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14041: skipped 000611 1788834628.002298 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x14041): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14041): Mirror 8: 0x89df76b6 (stale) Mirror 9: 0x89df76b6 (stale) Mirror 13: 0x89df76b6 (stale) Mirror 14: 0x89df76b6 (stale) Mirror 15: 0x89df76b6 (stale) CRC-32 checksum value for chunk [0, 0x14041): Mirror 1: 0x89df76b6 Mirror 16: 0x89df76b6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14041: skipped 000612 1788834629.138739 mapread 0x0050c7 thru 0x011d55 (0xcc8f bytes) 000615 1788834629.242529 read 0x002910 thru 0x007fed (0x56de bytes) 000617 1788834629.255435 mapread 0x01167a thru 0x014040 (0x29c7 bytes) 000619 1788834629.274601 mapread 0x009c0c thru 0x01264c (0x8a41 bytes) 000621 1788834629.295419 mapread 0x007316 thru 0x014040 (0xcd2b bytes) 000622 1788834629.345315 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x14041): Mirror 8: 0x89df76b6 (stale) Mirror 13: 0x89df76b6 (stale) Mirror 14: 0x89df76b6 (stale) Mirror 15: 0x89df76b6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14041): Mirror 8: 0x89df76b6 (stale) Mirror 13: 0x89df76b6 (stale) Mirror 14: 0x89df76b6 (stale) Mirror 15: 0x89df76b6 (stale) CRC-32 checksum value for chunk [0, 0x14041): Mirror 1: 0x89df76b6 Mirror 16: 0x89df76b6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14041: skipped 000624 1788834630.363605 mapread 0x00f6e0 thru 0x013995 (0x42b6 bytes) 000625 1788834630.402152 punch from 0x013c5c to 0x014040 (0x3e5 bytes) 000626 1788834630.465061 mapwrite 0x012ae2 thru 0x01bbad (0x90cc bytes) 000628 1788834630.627012 read 0x00ca6c thru 0x0143f1 (0x7986 bytes) 000630 1788834630.628096 read 0x00bd3f thru 0x0138dc (0x7b9e bytes) 000631 1788834630.631022 trunc from 0x01bbae to 0x006bb5 (0x14ff8 bytes) 000632 1788834630.655599 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x6bb6): Mirror 8: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6bb6): Mirror 8: 0x3042237a (stale) Mirror 13: 0x3042237a (stale) Mirror 14: 0x3042237a (stale) Mirror 15: 0x3042237a (stale) Mirror 16: 0x3042237a (stale) CRC-32 checksum value for chunk [0, 0x6bb6): Mirror 1: 0x3042237a Mirror 17: 0x3042237a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6bb6: skipped 000633 1788834631.776594 punch from 0x005dc1 to 0x006bb5 (0xdf5 bytes) 000635 1788834631.891640 mapwrite 0x037db4 thru 0x03a5d4 (0x2821 bytes) 000638 1788834632.015417 mapwrite 0x03319e thru 0x035324 (0x2187 bytes) 000639 1788834632.068870 mapread 0x01e215 thru 0x02ae8f (0xcc7b bytes) 000641 1788834632.088513 read 0x019a4b thru 0x0223cb (0x8981 bytes) 000642 1788834632.089150 read 0x00564c thru 0x006d69 (0x171e bytes) 000645 1788834632.100127 mapwrite 0x017a62 thru 0x026559 (0xeaf8 bytes) 000646 1788834632.156617 trunc from 0x03a5d5 to 0x022b62 (0x17a72 bytes) 000647 1788834632.182102 MIRROR_RESYNC 7 mirrors 000648 1788834633.601241 MIRROR_RESYNC 7 mirrors 000651 1788834634.069619 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) 000652 1788834634.444793 write 0x02a242 thru 0x02c69b (0x245a bytes) 000654 1788834634.602944 trunc from 0x02c69c to 0x024e7b (0x7820 bytes) 000655 1788834634.641148 write 0x034a5e thru 0x037ff6 (0x3599 bytes) 000656 1788834634.644624 write 0x0091cb thru 0x0190be (0xfef4 bytes) 000658 1788834634.661381 punch from 0x02e326 to 0x037ff6 (0x9cd1 bytes) 000659 1788834634.681410 mapread 0x00bd76 thru 0x00f288 (0x3513 bytes) 000662 1788834634.693548 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37ff7): Mirror 13: 0x605db995 (stale) Mirror 15: 0x605db995 (stale) Mirror 16: 0x605db995 (stale) Mirror 17: 0x605db995 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37ff7): Mirror 13: 0x605db995 (stale) Mirror 15: 0x605db995 (stale) Mirror 16: 0x605db995 (stale) Mirror 17: 0x605db995 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37ff7: skipped 000663 1788834635.174622 mapread 0x00e18d thru 0x015589 (0x73fd bytes) 000664 1788834635.198998 mapread 0x00958c thru 0x010e7b (0x78f0 bytes) 000665 1788834635.200786 write 0x02ea78 thru 0x035658 (0x6be1 bytes) 000666 1788834635.217273 read 0x0036ea thru 0x01263a (0xef51 bytes) 000668 1788834635.229493 mapread 0x00e746 thru 0x016f10 (0x87cb bytes) 000670 1788834635.231426 read 0x031f26 thru 0x037ff6 (0x60d1 bytes) 000671 1788834635.246406 MIRROR_RESYNC 5 mirrors 000672 1788834636.876480 trunc from 0x037ff7 to 0x0238bc (0x1473a bytes) 000673 1788834636.973527 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x238bd): Mirror 13: 0x9b61dce4 (stale) Mirror 15: 0x9b61dce4 (stale) Mirror 16: 0x9b61dce4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x238bd): Mirror 13: 0x9b61dce4 (stale) Mirror 15: 0x9b61dce4 (stale) Mirror 16: 0x9b61dce4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x238bd: skipped 000674 1788834637.393972 mapwrite 0x00a91b thru 0x0182a2 (0xd988 bytes) 000675 1788834637.500123 MIRROR_RESYNC 4 mirrors 000677 1788834638.441177 write 0x028158 thru 0x0326d6 (0xa57f bytes) 000678 1788834638.520194 punch from 0x024b91 to 0x02efd1 (0xa441 bytes) 000679 1788834638.541234 mapwrite 0x01eecc thru 0x02299c (0x3ad1 bytes) 000682 1788834638.652176 mapwrite 0x009fa6 thru 0x013b2b (0x9b86 bytes) 000683 1788834638.718816 trunc from 0x0326d7 to 0x022ad9 (0xfbfd bytes) 000684 1788834638.742373 mapread 0x01e0ec thru 0x01fe9a (0x1daf bytes) 000685 1788834638.744378 punch from 0x006c77 to 0x00b379 (0x4703 bytes) 000686 1788834638.750522 mapwrite 0x0123ed thru 0x01a111 (0x7d25 bytes) 000688 1788834638.810982 mapwrite 0x023ed9 thru 0x024258 (0x380 bytes) 000689 1788834638.878562 trunc from 0x024259 to 0x038afb (0x148a3 bytes) 000690 1788834638.900422 mapread 0x02990f thru 0x031e84 (0x8576 bytes) 000691 1788834638.912208 mapread 0x003be0 thru 0x007064 (0x3485 bytes) 000692 1788834638.923770 punch from 0x011361 to 0x017833 (0x64d3 bytes) 000695 1788834638.930934 write 0x00cb2c thru 0x01b232 (0xe707 bytes) 000697 1788834638.948364 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x38afc): Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38afc): Mirror 13: 0xf40e31dc (stale) Mirror 15: 0xf40e31dc (stale) Mirror 16: 0xf40e31dc (stale) CRC-32 checksum value for chunk [0, 0x38afc): Mirror 1: 0x91eb9d72 Mirror 17: 0x91eb9d72 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38afc: skipped 000698 1788834640.273400 mapread 0x015322 thru 0x015b06 (0x7e5 bytes) 000703 1788834640.359263 write 0x03ec90 thru 0x03ffff (0x1370 bytes) 000704 1788834640.497586 read 0x039eca thru 0x03c7ee (0x2925 bytes) 000706 1788834640.508137 read 0x0183c1 thru 0x020af0 (0x8730 bytes) 000707 1788834640.536393 read 0x02e189 thru 0x035be2 (0x7a5a bytes) 000709 1788834640.537669 mapread 0x0096ec thru 0x010227 (0x6b3c bytes) 000710 1788834640.542680 trunc from 0x040000 to 0x037f87 (0x8078 bytes) 000712 1788834640.581208 punch from 0x0075e4 to 0x007a01 (0x41e bytes) 000713 1788834640.600156 trunc from 0x037f88 to 0x025461 (0x12b26 bytes) 000714 1788834640.624881 mapwrite 0x026543 thru 0x029ada (0x3598 bytes) 000715 1788834640.696270 read 0x00ac89 thru 0x00d5b5 (0x292d bytes) 000716 1788834640.696957 read 0x01576e thru 0x019c0f (0x44a2 bytes) 000718 1788834640.698036 trunc from 0x029adb to 0x02d90b (0x3e31 bytes) 000719 1788834640.715365 mapwrite 0x0229a9 thru 0x023bce (0x1226 bytes) 000720 1788834640.754679 punch from 0x00e9ac to 0x0124be (0x3b13 bytes) 000721 1788834640.759089 punch from 0x0086f7 to 0x01282e (0xa138 bytes) 000722 1788834640.772021 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2d90c): Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d90c): Mirror 13: 0x2433361c (stale) Mirror 15: 0x2433361c (stale) Mirror 16: 0x2433361c (stale) Mirror 17: 0x56518e34 (stale) CRC-32 checksum value for chunk [0, 0x2d90c): Mirror 1: 0xc78a46b0 Mirror 18: 0xc78a46b0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d90c: skipped 000723 1788834641.946865 mapwrite 0x026d11 thru 0x0338b5 (0xcba5 bytes) 000724 1788834642.102505 MIRROR_RESYNC 6 mirrors 000727 1788834643.241464 trunc from 0x0338b6 to 0x0360bf (0x280a bytes) 000728 1788834643.345669 trunc from 0x0360c0 to 0x010eaa (0x25215 bytes) 000729 1788834643.395637 punch from 0x004a64 to 0x010eaa (0xc447 bytes) 000731 1788834643.402683 mapread 0x008d70 thru 0x00e3a8 (0x5639 bytes) 000732 1788834643.442564 trunc from 0x010eab to 0x00dd89 (0x3121 bytes) 000733 1788834643.468496 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0xdd8a): Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xdd8a): Mirror 13: 0xf776a13 (stale) Mirror 15: 0xf776a13 (stale) Mirror 16: 0xf776a13 (stale) Mirror 17: 0xf776a13 (stale) Mirror 18: 0xf776a13 (stale) CRC-32 checksum value for chunk [0, 0xdd8a): Mirror 1: 0xf776a13 Mirror 19: 0xf776a13 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xdd8a: skipped 000734 1788834644.537673 punch from 0x00c0f7 to 0x00dd89 (0x1c93 bytes) 000735 1788834644.639686 punch from 0x007a4a to 0x00d9d0 (0x5f87 bytes) 000737 1788834644.649073 mapwrite 0x017379 thru 0x025812 (0xe49a bytes) 000738 1788834644.819332 punch from 0x000f3a to 0x00de4a (0xcf11 bytes) 000739 1788834644.828746 mapread 0x010c5a thru 0x01c99a (0xbd41 bytes) 000740 1788834644.855029 MIRROR_RESYNC 7 mirrors 000741 1788834646.485469 write 0x028a1f thru 0x029c41 (0x1223 bytes) 000742 1788834646.598050 read 0x0150ef thru 0x021d49 (0xcc5b bytes) 000743 1788834646.620366 mapwrite 0x02e74b thru 0x0395b2 (0xae68 bytes) 000744 1788834646.799342 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x395b3): Mirror 13: 0xb55c755 (stale) Mirror 15: 0xb55c755 (stale) Mirror 17: 0xb55c755 (stale) Mirror 18: 0xb55c755 (stale) Mirror 19: 0xb55c755 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x395b3): Mirror 13: 0xb55c755 (stale) Mirror 15: 0xb55c755 (stale) Mirror 17: 0xb55c755 (stale) Mirror 18: 0xb55c755 (stale) Mirror 19: 0xb55c755 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x395b3: skipped 000745 1788834647.408398 read 0x036ed0 thru 0x0395b2 (0x26e3 bytes) 000746 1788834647.418936 mapwrite 0x039651 thru 0x03ffff (0x69af bytes) 000747 1788834647.550625 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xe7de4901 (stale) Mirror 15: 0xe7de4901 (stale) Mirror 17: 0xe7de4901 (stale) Mirror 18: 0xe7de4901 (stale) Mirror 19: 0xe7de4901 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x31f86194 Mirror 20: 0x31f86194 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000748 1788834648.464735 read 0x003def thru 0x005fa4 (0x21b6 bytes) 000749 1788834648.478665 punch from 0x039746 to 0x03fcea (0x65a5 bytes) 000750 1788834648.607166 write 0x021946 thru 0x02f97e (0xe039 bytes) 000753 1788834648.650885 MIRROR_RESYNC 7 mirrors 000754 1788834650.071251 MIRROR_RESYNC 7 mirrors 000755 1788834650.657475 punch from 0x0399ba to 0x03d7df (0x3e26 bytes) 000756 1788834650.730366 trunc from 0x040000 to 0x03a9da (0x5625 bytes) 000758 1788834650.768535 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a9db): Mirror 13: 0x478dd285 (stale) Mirror 15: 0x478dd285 (stale) Mirror 18: 0x478dd285 (stale) Mirror 19: 0x478dd285 (stale) Mirror 20: 0x478dd285 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a9db): Mirror 13: 0x478dd285 (stale) Mirror 15: 0x478dd285 (stale) Mirror 18: 0x478dd285 (stale) Mirror 19: 0x478dd285 (stale) Mirror 20: 0x478dd285 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a9db: skipped 000760 1788834651.401301 mapread 0x00fc73 thru 0x012ab8 (0x2e46 bytes) 000761 1788834651.435578 mapwrite 0x02f958 thru 0x037ec1 (0x856a bytes) 000763 1788834651.547270 MIRROR_RESYNC 6 mirrors 000764 1788834653.004010 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) 000765 1788834653.366262 MIRROR_EXTEND to 6 mirrors 000766 1788834654.304133 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) 000770 1788834654.881279 read 0x003608 thru 0x00d8bc (0xa2b5 bytes) 000771 1788834654.926989 MIRROR_EXTEND to 6 mirrors 000772 1788834655.904644 read 0x01f2d5 thru 0x022a23 (0x374f bytes) 000773 1788834655.928854 write 0x039e46 thru 0x03e35d (0x4518 bytes) 000775 1788834656.069447 mapread 0x017066 thru 0x0199cb (0x2966 bytes) 000776 1788834656.104361 punch from 0x015bb4 to 0x0202e7 (0xa734 bytes) 000779 1788834656.126599 mapread 0x02dcce thru 0x02fb0e (0x1e41 bytes) 000780 1788834656.142976 trunc from 0x03e35e to 0x01093a (0x2da23 bytes) 000781 1788834656.179044 punch from 0x005ec0 to 0x01093a (0xaa7b bytes) 000782 1788834656.185611 mapwrite 0x01a538 thru 0x01b402 (0xecb bytes) 000783 1788834656.311657 read 0x018ce5 thru 0x01b402 (0x271e bytes) 000784 1788834656.312412 mapwrite 0x0257de thru 0x02a63f (0x4e62 bytes) 000785 1788834656.384159 trunc from 0x02a640 to 0x01ee57 (0xb7e8 bytes) 000786 1788834656.403258 punch from 0x003652 to 0x0098d1 (0x6280 bytes) 000787 1788834656.407967 trunc from 0x01ee58 to 0x023e79 (0x5022 bytes) 000788 1788834656.425070 punch from 0x007aa0 to 0x00e38b (0x68ec bytes) 000789 1788834656.430994 mapread 0x0036e1 thru 0x004b75 (0x1495 bytes) 000790 1788834656.446071 write 0x025743 thru 0x02aa77 (0x5335 bytes) 000791 1788834656.449197 write 0x01a037 thru 0x01d862 (0x382c bytes) 000792 1788834656.451292 read 0x01e932 thru 0x027e8a (0x9559 bytes) 000793 1788834656.482740 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2aa78): Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2aa78): Mirror 18: 0x5849fb88 (stale) Mirror 19: 0x5849fb88 (stale) Mirror 20: 0x5849fb88 (stale) Mirror 21: 0x5849fb88 (stale) Mirror 22: 0x5849fb88 (stale) CRC-32 checksum value for chunk [0, 0x2aa78): Mirror 1: 0x1625f3f2 Mirror 23: 0x1625f3f2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2aa78: skipped 000794 1788834657.624905 punch from 0x01ab21 to 0x01e5bf (0x3a9f bytes) 000795 1788834657.721578 mapread 0x024199 thru 0x027a95 (0x38fd bytes) 000797 1788834657.748685 write 0x00f0fa thru 0x016861 (0x7768 bytes) 000798 1788834657.762137 read 0x00d725 thru 0x01500c (0x78e8 bytes) 000799 1788834657.773011 mapwrite 0x02617b thru 0x03447a (0xe300 bytes) 000801 1788834657.948288 read 0x00d01f thru 0x00e0a1 (0x1083 bytes) 000803 1788834657.952172 write 0x03d568 thru 0x03e913 (0x13ac bytes) 000804 1788834657.954791 write 0x004deb thru 0x00e29c (0x94b2 bytes) 000807 1788834657.990496 punch from 0x011ec4 to 0x01222b (0x368 bytes) 000809 1788834658.015252 write 0x00ade1 thru 0x00b206 (0x426 bytes) 000811 1788834658.018051 trunc from 0x03e914 to 0x02d1dd (0x11736 bytes) 000812 1788834658.040179 mapread 0x013a76 thru 0x01c759 (0x8ce4 bytes) 000813 1788834658.043424 trunc from 0x02d1de to 0x01ce7c (0x10361 bytes) 000814 1788834658.076346 read 0x01c900 thru 0x01ce7c (0x57d bytes) 000815 1788834658.076975 punch from 0x0130cd to 0x01ce7c (0x9db0 bytes) 000816 1788834658.085681 trunc from 0x01ce7d to 0x0386a3 (0x1b827 bytes) 000818 1788834658.110478 write 0x02f6dc thru 0x03289c (0x31c1 bytes) 000820 1788834658.123061 mapwrite 0x02712e thru 0x033930 (0xc803 bytes) 000823 1788834658.188763 mapwrite 0x02c08d thru 0x03011a (0x408e bytes) 000825 1788834658.259392 mapwrite 0x020888 thru 0x02aca4 (0xa41d bytes) 000826 1788834658.361106 trunc from 0x0386a4 to 0x01fc38 (0x18a6b bytes) 000827 1788834658.386682 punch from 0x00d653 to 0x0138bc (0x626a bytes) 000828 1788834658.393017 mapread 0x00ef93 thru 0x01e16b (0xf1d9 bytes) 000829 1788834658.422370 punch from 0x0104c0 to 0x019788 (0x92c9 bytes) 000831 1788834658.427758 trunc from 0x01fc39 to 0x03e06f (0x1e437 bytes) 000832 1788834658.447259 write 0x02e814 thru 0x03aeb5 (0xc6a2 bytes) 000834 1788834658.465951 MIRROR_RESYNC 7 mirrors 000837 1788834659.779740 read 0x015f28 thru 0x016e7d (0xf56 bytes) 000838 1788834659.789441 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) 000840 1788834660.210286 punch from 0x02ae98 to 0x039798 (0xe901 bytes) 000841 1788834660.287778 mapwrite 0x017b7e thru 0x01ffd8 (0x845b bytes) 000842 1788834660.366958 trunc from 0x03e070 to 0x03974c (0x4923 bytes) 000843 1788834660.392356 punch from 0x01ed1a to 0x029172 (0xa459 bytes) 000845 1788834660.401393 punch from 0x004a6a to 0x006696 (0x1c2d bytes) 000846 1788834660.409197 mapread 0x011132 thru 0x012963 (0x1832 bytes) 000847 1788834660.432188 punch from 0x000f5d to 0x00ed8b (0xde2f bytes) 000849 1788834660.440635 trunc from 0x03974d to 0x0098dd (0x2fe6f bytes) 000850 1788834660.466459 punch from 0x000774 to 0x0063ec (0x5c79 bytes) 000851 1788834660.471574 trunc from 0x0098de to 0x01110d (0x7830 bytes) 000852 1788834660.492590 read 0x007fef thru 0x01110d (0x911f bytes) 000856 1788834660.509284 write 0x026c0a thru 0x027a45 (0xe3c bytes) 000859 1788834660.511933 punch from 0x006427 to 0x013742 (0xd31c bytes) 000860 1788834660.522348 read 0x01edf5 thru 0x024238 (0x5444 bytes) truncating to largest ever: 0x3fe3c 000864 1788834660.539492 trunc from 0x027a46 to 0x03fe3b (0x183f6 bytes) 000865 1788834660.562111 trunc from 0x03fe3c to 0x0302bb (0xfb80 bytes) 000866 1788834660.580437 punch from 0x0219cf to 0x026757 (0x4d89 bytes) 000867 1788834660.586202 write 0x02fa4c thru 0x034d73 (0x5328 bytes) 000868 1788834660.597324 mapwrite 0x0334ab thru 0x03f034 (0xbb8a bytes) 000869 1788834660.730770 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3f035): Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f035): Mirror 19: 0x61056053 (stale) Mirror 20: 0x61056053 (stale) Mirror 21: 0x61056053 (stale) Mirror 22: 0x61056053 (stale) Mirror 23: 0x61056053 (stale) CRC-32 checksum value for chunk [0, 0x3f035): Mirror 1: 0x65a055c6 Mirror 24: 0x65a055c6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f035: skipped 000871 1788834661.649347 MIRROR_RESYNC 7 mirrors 000872 1788834662.842559 read 0x0103a7 thru 0x0167ce (0x6428 bytes) 000873 1788834662.853811 read 0x023921 thru 0x02f1c3 (0xb8a3 bytes) 000874 1788834662.869927 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) 000875 1788834663.405936 write 0x03b77d thru 0x03ffff (0x4883 bytes) 000876 1788834663.488919 trunc from 0x040000 to 0x02caf0 (0x1350f bytes) 000878 1788834663.511274 write 0x038231 thru 0x03a903 (0x26d3 bytes) 000880 1788834663.513330 write 0x000e63 thru 0x001c9b (0xe39 bytes) 000881 1788834663.523363 trunc from 0x03a904 to 0x007811 (0x330f2 bytes) 000883 1788834663.550078 write 0x006613 thru 0x012235 (0xbc23 bytes) 000885 1788834663.564766 trunc from 0x012236 to 0x0017e6 (0x10a4f bytes) 000887 1788834663.592505 read 0x000933 thru 0x0017e6 (0xeb4 bytes) 000890 1788834663.598360 punch from 0x0002eb to 0x0017e6 (0x14fc bytes) 000892 1788834663.620903 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x17e7): Mirror 20: 0xda78eda8 (stale) Mirror 21: 0xda78eda8 (stale) Mirror 22: 0xda78eda8 (stale) Mirror 24: 0xda78eda8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17e7): Mirror 20: 0xda78eda8 (stale) Mirror 21: 0xda78eda8 (stale) Mirror 22: 0xda78eda8 (stale) Mirror 24: 0xda78eda8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17e7: skipped 000893 1788834664.020175 trunc from 0x0017e7 to 0x0164a1 (0x14cbb bytes) 000894 1788834664.043384 mapwrite 0x026792 thru 0x0271c6 (0xa35 bytes) 000895 1788834664.141302 MIRROR_RESYNC 5 mirrors 000896 1788834665.738268 read 0x0248fd thru 0x0271c6 (0x28ca bytes) 000897 1788834665.753725 mapwrite 0x0229b1 thru 0x02a7f2 (0x7e42 bytes) 000899 1788834665.898394 mapread 0x0006e4 thru 0x00dcd7 (0xd5f4 bytes) 000900 1788834665.921308 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2a7f3): Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a7f3): Mirror 20: 0x6f6bdb10 (stale) Mirror 21: 0x6f6bdb10 (stale) Mirror 22: 0x6f6bdb10 (stale) Mirror 24: 0x6f6bdb10 (stale) CRC-32 checksum value for chunk [0, 0x2a7f3): Mirror 1: 0x7b04d820 Mirror 25: 0x7b04d820 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a7f3: skipped 000901 1788834667.088892 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a7f3): Mirror 20: 0x6f6bdb10 (stale) Mirror 21: 0x6f6bdb10 (stale) Mirror 24: 0x6f6bdb10 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a7f3): Mirror 20: 0x6f6bdb10 (stale) Mirror 21: 0x6f6bdb10 (stale) Mirror 24: 0x6f6bdb10 (stale) CRC-32 checksum value for chunk [0, 0x2a7f3): Mirror 1: 0x7b04d820 Mirror 25: 0x7b04d820 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a7f3: skipped 000902 1788834667.878909 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a7f3): Mirror 21: 0x6f6bdb10 (stale) Mirror 24: 0x6f6bdb10 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a7f3): Mirror 21: 0x6f6bdb10 (stale) Mirror 24: 0x6f6bdb10 (stale) CRC-32 checksum value for chunk [0, 0x2a7f3): Mirror 1: 0x7b04d820 Mirror 25: 0x7b04d820 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a7f3: skipped 000903 1788834668.644089 trunc from 0x02a7f3 to 0x01a0e7 (0x1070b bytes) 000908 1788834668.764147 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a0e8): Mirror 21: 0x212f2dd9 (stale) Mirror 24: 0x212f2dd9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a0e8): Mirror 21: 0x212f2dd9 (stale) Mirror 24: 0x212f2dd9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a0e8: skipped 000910 1788834669.107066 mapwrite 0x03352f thru 0x03e9d8 (0xb4aa bytes) 000913 1788834669.200569 write 0x018039 thru 0x01a4fb (0x24c3 bytes) 000915 1788834669.221038 write 0x03636e thru 0x0389cb (0x265e bytes) 000917 1788834669.232206 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e9d9): Mirror 21: 0x379476cc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e9d9): Mirror 21: 0x379476cc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e9d9: skipped 000918 1788834669.577246 MIRROR_RESYNC 2 mirrors 000919 1788834670.482937 mapread 0x02cbbc thru 0x0324cb (0x5910 bytes) 000920 1788834670.517394 read 0x028e9a thru 0x02ae36 (0x1f9d bytes) 000921 1788834670.517840 write 0x02656b thru 0x02beba (0x5950 bytes) 000922 1788834670.602382 punch from 0x01215c to 0x018bbd (0x6a62 bytes) 000923 1788834670.616244 mapread 0x03e2f1 thru 0x03e9d8 (0x6e8 bytes) 000924 1788834670.629167 mapread 0x01c997 thru 0x01d8fc (0xf66 bytes) 000927 1788834670.640036 mapwrite 0x01fbb2 thru 0x02ea75 (0xeec4 bytes) 000928 1788834670.732046 write 0x02754f thru 0x02b17e (0x3c30 bytes) 000930 1788834670.734392 mapread 0x02915b thru 0x02fd09 (0x6baf bytes) 000931 1788834670.736326 read 0x01c223 thru 0x021bae (0x598c bytes) 000934 1788834670.738355 punch from 0x013077 to 0x01b22c (0x81b6 bytes) 000935 1788834670.752237 trunc from 0x03e9d9 to 0x028c9d (0x15d3b bytes) 000936 1788834670.783609 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) 000937 1788834670.989673 punch from 0x02190f to 0x022b71 (0x1263 bytes) 000938 1788834670.997634 trunc from 0x028c9e to 0x02545c (0x3841 bytes) 000939 1788834671.018482 write 0x020abe thru 0x02bc30 (0xb173 bytes) 000941 1788834671.027482 punch from 0x00b169 to 0x0195fe (0xe496 bytes) 000943 1788834671.050941 read 0x004b3e thru 0x008614 (0x3ad7 bytes) 000944 1788834671.062140 read 0x01c4cf thru 0x02b0ef (0xec21 bytes) 000946 1788834671.070285 mapwrite 0x011a63 thru 0x021405 (0xf9a3 bytes) 000947 1788834671.129867 mapwrite 0x001edd thru 0x0115e5 (0xf709 bytes) 000950 1788834671.190691 mapread 0x01c770 thru 0x026d72 (0xa603 bytes) 000951 1788834671.195959 read 0x00b9e4 thru 0x00e10e (0x272b bytes) 000953 1788834671.197318 mapread 0x024be0 thru 0x02bc30 (0x7051 bytes) 000954 1788834671.198149 read 0x00bc5e thru 0x011d0a (0x60ad bytes) 000956 1788834671.204926 write 0x03291e thru 0x033968 (0x104b bytes) 000957 1788834671.207142 mapwrite 0x019fc3 thru 0x026421 (0xc45f bytes) 000960 1788834671.261471 trunc from 0x033969 to 0x020f95 (0x129d3 bytes) 000962 1788834671.279070 mapwrite 0x02aa80 thru 0x02c590 (0x1b11 bytes) 000965 1788834671.351038 write 0x00b08b thru 0x01917f (0xe0f5 bytes) 000967 1788834671.353322 mapwrite 0x0205ac thru 0x02db5f (0xd5b4 bytes) 000968 1788834671.416028 mapwrite 0x00b132 thru 0x011c11 (0x6ae0 bytes) 000971 1788834671.471630 read 0x020009 thru 0x029588 (0x9580 bytes) 000972 1788834671.473043 mapwrite 0x004f28 thru 0x014bcf (0xfca8 bytes) 000973 1788834671.544899 punch from 0x021368 to 0x02db5f (0xc7f8 bytes) 000975 1788834671.550232 mapread 0x011566 thru 0x020437 (0xeed2 bytes) 000977 1788834671.552161 mapread 0x003faa thru 0x007e70 (0x3ec7 bytes) 000978 1788834671.554136 read 0x005270 thru 0x009ba2 (0x4933 bytes) 000984 1788834671.560329 punch from 0x024251 to 0x02c02d (0x7ddd bytes) 000986 1788834671.567021 punch from 0x02b8a7 to 0x02db5f (0x22b9 bytes) 000988 1788834671.572682 read 0x0137b1 thru 0x01f0b1 (0xb901 bytes) 000989 1788834671.573349 trunc from 0x02db60 to 0x03e46b (0x1090c bytes) 000990 1788834671.594610 write 0x024cf5 thru 0x027b35 (0x2e41 bytes) 000992 1788834671.609395 punch from 0x022631 to 0x02cee6 (0xa8b6 bytes) 000993 1788834671.628234 read 0x006d1e thru 0x013977 (0xcc5a bytes) 000994 1788834671.629169 punch from 0x0262cd to 0x029096 (0x2dca bytes) 000995 1788834671.634005 mapwrite 0x039243 thru 0x03ffff (0x6dbd bytes) 000996 1788834671.754115 mapread 0x01fdd7 thru 0x02d325 (0xd54f bytes) 000997 1788834671.755207 mapwrite 0x02d3db thru 0x02ffcd (0x2bf3 bytes) 000998 1788834671.806149 mapwrite 0x020697 thru 0x0258c6 (0x5230 bytes) 000999 1788834671.859052 trunc from 0x040000 to 0x01fb7f (0x20480 bytes) 001000 1788834671.880512 write 0x00c69d thru 0x0163e6 (0x9d4a bytes) All operations completed A-OK! PASS 70a (121s) == sanity-flr test 71: check mirror extend parent fid ==== 22:31:16 (1788834676) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 22:31:20 (1788834680) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.280945 s, 7.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133375 s, 7.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 22:31:28 (1788834688) Starting client: oleg651-client.virtnet: -o user_xattr,flock 192.168.206.151@tcp:/lustre /mnt/lustre2 Starting client: oleg651-client.virtnet: -o user_xattr,flock 192.168.206.151@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 1094592 .. Extending file size to 7148448 .. failed Extending file size to 7567648 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7791776 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8199136 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8420992 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7570976 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8288992 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9008224 .. Extending file size to 9358496 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57408 57409 57411 57413 57414 done 192.168.206.151@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg651-client.virtnet /mnt/lustre2 (opts:) 192.168.206.151@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg651-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (80s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 22:32:48 (1788834768) Starting client: oleg651-client.virtnet: -o user_xattr,flock 192.168.206.151@tcp:/lustre /mnt/lustre2 Starting client: oleg651-client.virtnet: -o user_xattr,flock 192.168.206.151@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 4453888 .. Extending file size to 8442880 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 5804704 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6053152 .. Extending file size to 8448608 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8416512 .. Extending file size to 8805856 .. Extending file size to 9176416 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9196192 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58896 58897 58899 58901 58902 done 192.168.206.151@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg651-client.virtnet /mnt/lustre2 (opts:) 192.168.206.151@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg651-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (86s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 22:34:14 (1788834854) Starting client: oleg651-client.virtnet: -o user_xattr,flock 192.168.206.151@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.112576 s, 18.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.268429 s, 7.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xe911a30 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.206.151@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg651-client.virtnet /mnt/lustre2 (opts:) PASS 200c (17s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 22:34:33 (1788834873) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.089443 s, 23.4 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 22:34:40 (1788834880) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ff:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ec:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ec:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ed:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x308:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 22:34:47 (1788834887) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.236691 s, 8.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ee:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x309:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 22:34:56 (1788834896) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.20466 s, 10.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ef:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ef:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30c:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 22:35:04 (1788834904) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.372833 s, 5.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30e:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 22:35:38 (1788834938) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg651-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105823 s, 9.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f2:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x310:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f2:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x310:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f3:0x0] } PASS 204d (65s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 22:36:44 (1788835004) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg651-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x311:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x312:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.456568 s, 4.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x311:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x312:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x311:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x312:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (61s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 22:37:44 (1788835064) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg651-server mds-ost sync done. Creating new pool test_204f oleg651-server: Pool lustre.test_204f created Adding targets to pool oleg651-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg651-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.229027 s, 9.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x313:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x313:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg651-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg651-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg651-server: Pool lustre.test_204f destroyed PASS 204f (81s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 22:39:05 (1788835145) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 22:39:13 (1788835153) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 22:39:22 (1788835162) oleg651-server: Pool lustre.test_206 created Adding targets to pool oleg651-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg651-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg651-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg651-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg651-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 22:39:47 (1788835187) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421024 s, 24.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350749 s, 29.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x193931f9 Mirror 3: 0x193931f9 PASS 207 (9s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 22:39:56 (1788835196) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (5s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 22:40:01 (1788835201) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 22:40:05 (1788835205) Starting client: oleg651-client.virtnet: -o user_xattr,flock 192.168.206.151@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.148671 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1e70000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xc10000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xc10000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xc10000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xf20000 Read this (4 bytes): BBBB 192.168.206.151@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg651-client.virtnet /mnt/lustre2 (opts:) PASS 209a (36s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 22:40:41 (1788835241) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0260573 s, 314 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 22:40:49 (1788835249) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667418 s, 15.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x224 lmm_fid: [0x200000404:0x224:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x224 lmm_fid: [0x200000404:0x224:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x300:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834468 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 22:40:51.000000000 -0400 Modify: 2026-09-07 22:40:51.000000000 -0400 Change: 2026-09-07 22:40:53.000000000 -0400 Birth: 2026-09-07 22:40:52.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 22:40:59 (1788835259) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 22:41:03 (1788835263) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.534447 s, 76.6 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0825936 s, 49.6 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 7 22:41 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x226:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 7 22:41 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 22:41:13 (1788835273) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 22:41:20 (1788835280) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1730 sec =========== 22:41:29 (1788835289) === sanity-flr: start cleanup 22:41:30 (1788835290) === === sanity-flr: finish cleanup 22:41:37 (1788835297) ===