-----============= acceptance-small: sanity-flr ============----- Sat Sep 5 03:39:46 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg359-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 03:40:04 (1788594004) === oleg359-client.virtnet: executing check_config_client /mnt/lustre oleg359-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg359-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9ece614a8000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9ece614a8000.idle_timeout=debug disable quota as required oleg359-server: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 03:40:36 (1788594036) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7654] == sanity-flr test 0a: lfs mirror create with -N option == 03:40:38 (1788594038) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 03:40:56 (1788594056) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 03:40:59 (1788594059) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 03:41:03 (1788594063) PASS 0d (23s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 03:41:26 (1788594086) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 03:41:29 (1788594089) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 03:41:33 (1788594093) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9ece614a8000.stats=clear osc.lustre-OST0001-osc-ffff9ece614a8000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 03:41:41 (1788594101) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 03:41:52 (1788594112) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 03:41:59 (1788594119) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 03:42:03 (1788594123) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 03:42:12 (1788594132) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 03:42:22 (1788594142) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 03:42:29 (1788594149) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 03:42:33 (1788594153) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 03:42:41 (1788594161) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 03:42:52 (1788594172) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 03:42:58 (1788594178) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 1.02697 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130842 s, 8.0 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 03:43:08 (1788594188) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.378679 s, 33.2 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 03:43:16 (1788594196) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 03:43:25 (1788594205) Stopping /mnt/lustre-ost1 (opts:) on oleg359-server oleg359-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ece614a8000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ece614a8000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg359-server: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg359-server: oleg359-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg359-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg359-server oleg359-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9ece614a8000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ece614a8000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg359-server: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg359-server: oleg359-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg359-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (77s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 03:44:43 (1788594283) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.605757 s, 19.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg359-server oleg359-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ece614a8000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ece614a8000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg359-server: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg359-server: oleg359-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg359-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (45s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 03:45:27 (1788594327) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg359-server oleg359-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ece614a8000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ece614a8000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg359-server: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg359-server: oleg359-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg359-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg359-server oleg359-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9ece614a8000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ece614a8000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg359-server: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg359-server: oleg359-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg359-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (83s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 03:46:50 (1788594410) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 6.352 s, 8.3 MB/s -rw-r--r-- 1 root root 52428800 Sep 5 03:46 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.214s user 0m0.001s sys 0m0.164s Stopping /mnt/lustre-ost1 (opts:) on oleg359-server oleg359-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ece614a8000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ece614a8000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.980s user 0m0.001s sys 0m0.177s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg359-server: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg359-server: oleg359-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg359-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg359-server oleg359-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9ece614a8000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ece614a8000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.638s user 0m0.000s sys 0m0.187s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg359-server: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg359-server: oleg359-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg359-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg359-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9ece614a8000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ece614a8000.ost_server_uuid in FULL state after 0 sec PASS 33b (93s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 03:48:23 (1788594503) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 03:48:26 (1788594506) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 03:48:29 (1788594509) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 03:48:33 (1788594513) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 03:48:40 (1788594520) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.530791 s, 23.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 03:48:52 (1788594532) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176022 s, 59.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398449 s, 26.3 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 03:48:59 (1788594539) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 03:49:07 (1788594547) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00474405 s, 216 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00143805 s, 712 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000836814 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00175255 s, 584 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 03:49:18 (1788594558) create test files with size 15732117 .. aa5a1ba456adecf46839fc4edefecb66 - d333b5856c2c3fb1f1c2ac76def3d803 - 8d16937a4ec6fbbff61539c7d526938c - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg359-client.virtnet /mnt/lustre (opts:) Starting client: oleg359-client.virtnet: -o user_xattr,flock 192.168.203.159@tcp:/lustre /mnt/lustre PASS 37 (26s) == sanity-flr test 38: resync ============================ 03:49:44 (1788594584) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 03:49:57 (1788594597) PASS 39 (5s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 03:50:03 (1788594603) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121742 s, 8.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xba:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386171 s, 27.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xda:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 03:50:11 (1788594611) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0791235 s, 26.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.392223 s, 10.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.196195 s, 10.7 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (14s) == sanity-flr test 42: lfs mirror verify ================= 03:50:25 (1788594625) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 03:50:29 (1788594629) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 03:50:32 (1788594632) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576959 s, 18.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 03:50:39 (1788594639) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 03:50:43 (1788594643) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe1:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc3:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc3:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 03:50:53 (1788594653) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.370056 s, 28.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg359-client.virtnet /mnt/lustre (opts:) Starting client: oleg359-client.virtnet: -o user_xattr,flock 192.168.203.159@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 03:51:06 (1788594666) llite.lustre-ffff9ece4f80a800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.330193 s, 31.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg359-client.virtnet /mnt/lustre (opts:) Starting client: oleg359-client.virtnet: -o user_xattr,flock 192.168.203.159@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9ece51039000.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 03:51:21 (1788594681) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (38s) == sanity-flr test 46: Verify setstripe --copy option ==== 03:51:59 (1788594719) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (35s) == sanity-flr test 47: Verify mirror obj alloc =========== 03:52:34 (1788594754) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 03:52:38 (1788594758) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.263781 s, 11.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.15296 s, 20.6 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xba059108 (nosync,stale) Mirror 2: 0xba059108 (nosync,stale) ** verify mirror 2 doesn't change original checksum: a8f5b4cb8272d0466e80c8fd13799bd9 - mirror 1 checksum: ad0c31f4e73428df33f62769e8980ecb - mirror 2 checksum: a8f5b4cb8272d0466e80c8fd13799bd9 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-05T03:52:41-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-05T03:52:41-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd9:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: a8f5b4cb8272d0466e80c8fd13799bd9 - mirror 1 checksum: ad0c31f4e73428df33f62769e8980ecb - mirror 2 checksum: ad0c31f4e73428df33f62769e8980ecb - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-05T03:52:41-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-05T03:52:41-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd9:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 03:52:50 (1788594770) Failing mds1 on oleg359-server Stopping /mnt/lustre-mds1 (opts:) on oleg359-server 03:53:02 (1788594782) shut down facet: mds1 facet_host: oleg359-server facet_failover_host: oleg359-server Failover mds1 to oleg359-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg359-server: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-MDT0000 03:53:35 (1788594815) targets are mounted 03:53:35 (1788594815) facet_failover done oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308558 s, 34.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } PASS 50A (76s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 03:54:06 (1788594846) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 03:54:10 (1788594850) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 03:54:13 (1788594853) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00516586 s, 3.2 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 03:54:20 (1788594860) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 03:54:24 (1788594864) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0140392 s, 292 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0923048 s, 44.4 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 03:54:32 (1788594872) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (21s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 03:54:53 (1788594893) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 03:55:22 (1788594922) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 03:55:45 (1788594945) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 03:56:09 (1788594969) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (6s) == sanity-flr test 70: mirror create and split race ====== 03:56:16 (1788594976) mirror create pid 54352 mirror split pid 54353 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54352 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54353 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 03:57:23 (1788595043) Seed set to 5045 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read 000003 1788595045.877172 write 0x03e38c thru 0x03ffff (0x1c74 bytes) 000005 1788595045.878931 read 0x02e5c9 thru 0x038182 (0x9bba bytes) 000006 1788595045.890444 mapwrite 0x02eba4 thru 0x031aba (0x2f17 bytes) 000007 1788595046.018315 mapwrite 0x01abcb thru 0x02557b (0xa9b1 bytes) 000008 1788595046.166743 write 0x03b8d4 thru 0x03ffff (0x472c bytes) 000010 1788595046.195923 write 0x032180 thru 0x03402a (0x1eab bytes) 000011 1788595046.198876 punch from 0x01fe10 to 0x02b133 (0xb324 bytes) truncating to largest ever: 0x2c0f4 000012 1788595046.219276 trunc from 0x040000 to 0x02c0f3 (0x13f0c bytes) 000014 1788595046.235928 punch from 0x0063ab to 0x00ca6e (0x66c4 bytes) 000015 1788595046.243443 mapread 0x023cc3 thru 0x02c0f3 (0x8431 bytes) 000017 1788595046.258720 trunc from 0x02c0f4 to 0x00bd12 (0x203e1 bytes) 000018 1788595046.286476 punch from 0x0031e0 to 0x00bd12 (0x8b33 bytes) 000019 1788595046.291590 punch from 0x00328a to 0x00a78f (0x7506 bytes) 000020 1788595046.296229 write 0x00bbba thru 0x011e4f (0x6296 bytes) 000021 1788595046.303492 trunc from 0x011e50 to 0x012dfc (0xfad bytes) 000023 1788595046.329853 MIRROR_EXTEND to 2 mirrors 000024 1788595047.464336 mapwrite 0x0292ce thru 0x038db6 (0xfae9 bytes) 000025 1788595047.745466 write 0x02e3e2 thru 0x03b795 (0xd3b4 bytes) 000027 1788595047.755203 mapread 0x038679 thru 0x03b795 (0x311d bytes) 000028 1788595047.756257 write 0x0261a8 thru 0x02e4ad (0x8306 bytes) 000029 1788595047.770558 read 0x011093 thru 0x015b20 (0x4a8e bytes) 000030 1788595047.792089 write 0x00dfcf thru 0x018ae8 (0xab1a bytes) 000031 1788595047.810260 punch from 0x0207a8 to 0x02d3f6 (0xcc4f bytes) 000032 1788595047.845325 mapread 0x00f01d thru 0x01cd2f (0xdd13 bytes) truncating to largest ever: 0x346cf 000033 1788595047.869316 trunc from 0x03b796 to 0x0346ce (0x70c7 bytes) 000034 1788595047.903343 write 0x0177c9 thru 0x01d3b3 (0x5beb bytes) 000035 1788595047.910611 trunc from 0x0346cf to 0x00a409 (0x2a2c5 bytes) 000036 1788595047.945614 read 0x000f9a thru 0x006951 (0x59b8 bytes) truncating to largest ever: 0x3a10f 000037 1788595047.965176 trunc from 0x00a40a to 0x03a10e (0x2fd05 bytes) 000038 1788595048.002352 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ad:0x0]: Invalid argument (22) 000039 1788595048.204515 trunc from 0x03a10f to 0x00aff3 (0x2f11b bytes) 000040 1788595048.240598 trunc from 0x00aff4 to 0x02b48c (0x20499 bytes) 000042 1788595048.265112 mapread 0x00fab4 thru 0x0191a7 (0x96f4 bytes) 000043 1788595048.317556 write 0x024cc1 thru 0x027801 (0x2b41 bytes) 000044 1788595048.348220 MIRROR_EXTEND to 2 mirrors 000045 1788595049.481535 mapwrite 0x01bb76 thru 0x027bbc (0xc047 bytes) 000049 1788595049.726238 punch from 0x0087d9 to 0x017f86 (0xf7ae bytes) 000050 1788595049.731868 mapread 0x010fb3 thru 0x01e0bc (0xd10a bytes) 000051 1788595049.740184 read 0x00cbba thru 0x012230 (0x5677 bytes) 000052 1788595049.741593 mapwrite 0x0111bd thru 0x01ed7d (0xdbc1 bytes) 000054 1788595049.795195 mapread 0x005d0d thru 0x01498a (0xec7e bytes) 000055 1788595049.804166 punch from 0x02622f to 0x02b48c (0x525e bytes) 000056 1788595049.812343 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ad:0x0]: Invalid argument (22) 000058 1788595049.940608 trunc from 0x02b48d to 0x011bfc (0x19890 bytes) 000059 1788595049.981558 punch from 0x0021d6 to 0x009e1a (0x7c45 bytes) 000060 1788595049.990970 write 0x018fd4 thru 0x01d41f (0x444c bytes) 000061 1788595049.995041 punch from 0x009ade to 0x00ca10 (0x2f33 bytes) 000063 1788595050.014134 write 0x01b092 thru 0x01f567 (0x44d6 bytes) 000067 1788595050.017284 mapwrite 0x00da94 thru 0x017a22 (0x9f8f bytes) 000068 1788595050.119329 mapread 0x0158fc thru 0x01aa86 (0x518b bytes) 000070 1788595050.121630 read 0x00b4fe thru 0x01a72d (0xf230 bytes) 000071 1788595050.124128 write 0x00381a thru 0x012b2b (0xf312 bytes) 000072 1788595050.138191 mapread 0x007ccc thru 0x0169e1 (0xed16 bytes) 000074 1788595050.142825 write 0x000b0f thru 0x006e55 (0x6347 bytes) 000076 1788595050.152964 trunc from 0x01f568 to 0x02d40f (0xdea8 bytes) 000077 1788595050.172011 read 0x01c691 thru 0x01ed1e (0x268e bytes) 000078 1788595050.173071 punch from 0x01f9a7 to 0x02bd79 (0xc3d3 bytes) 000079 1788595050.193584 punch from 0x009d6b to 0x01084e (0x6ae4 bytes) 000082 1788595050.208530 read 0x020e48 thru 0x02d40f (0xc5c8 bytes) 000084 1788595050.224722 mapwrite 0x0045ea thru 0x010bc6 (0xc5dd bytes) 000087 1788595050.281206 trunc from 0x02d410 to 0x02e218 (0xe09 bytes) 000089 1788595050.302939 read 0x02cdcf thru 0x02dfce (0x1200 bytes) 000090 1788595050.303771 write 0x037239 thru 0x03a654 (0x341c bytes) 000091 1788595050.305656 trunc from 0x03a655 to 0x002320 (0x38334 bytes) 000093 1788595050.325987 mapread 0x000dc5 thru 0x002320 (0x155c bytes) 000094 1788595050.327075 read 0x000ce7 thru 0x002320 (0x163a bytes) 000096 1788595050.329284 write 0x0316c5 thru 0x036479 (0x4db5 bytes) 000097 1788595050.331225 punch from 0x01100c to 0x01490c (0x3901 bytes) 000099 1788595050.346579 trunc from 0x03647a to 0x032764 (0x3d15 bytes) 000100 1788595050.363076 trunc from 0x032765 to 0x01f09e (0x136c6 bytes) 000102 1788595050.379214 punch from 0x007503 to 0x00b8fe (0x43fc bytes) 000104 1788595050.384122 read 0x00a954 thru 0x00db98 (0x3245 bytes) 000105 1788595050.394004 write 0x0300d8 thru 0x03ff46 (0xfe6f bytes) 000106 1788595050.395937 write 0x03df8f thru 0x03ffff (0x2071 bytes) 000107 1788595050.396814 mapwrite 0x01a3c6 thru 0x02290c (0x8547 bytes) 000108 1788595050.445526 write 0x01cd58 thru 0x02019b (0x3444 bytes) 000109 1788595050.446527 write 0x02c726 thru 0x035f84 (0x985f bytes) 000111 1788595050.448891 mapwrite 0x010077 thru 0x01ea1a (0xe9a4 bytes) 000112 1788595050.538174 trunc from 0x040000 to 0x01461e (0x2b9e1 bytes) 000113 1788595050.567818 mapwrite 0x01c968 thru 0x025158 (0x87f1 bytes) 000114 1788595050.751484 mapread 0x009c2f thru 0x0181d2 (0xe5a4 bytes) 000115 1788595050.785133 mapwrite 0x018602 thru 0x01f037 (0x6a36 bytes) 000116 1788595050.865201 punch from 0x01e1ee to 0x025158 (0x6f6b bytes) 000118 1788595050.870729 trunc from 0x025159 to 0x0214e0 (0x3c78 bytes) 000120 1788595050.889796 read 0x01de6c thru 0x0214e0 (0x3675 bytes) 000122 1788595050.898938 mapwrite 0x014ee6 thru 0x016a19 (0x1b34 bytes) 000124 1788595050.951085 mapwrite 0x02edfa thru 0x02ff52 (0x1159 bytes) 000126 1788595051.020585 mapwrite 0x0346e0 thru 0x035ef0 (0x1811 bytes) 000127 1788595051.094148 mapread 0x01068f thru 0x01d283 (0xcbf5 bytes) 000128 1788595051.096173 read 0x0140e5 thru 0x01cbe3 (0x8aff bytes) 000129 1788595051.096865 trunc from 0x035ef1 to 0x02a743 (0xb7ad bytes) 000133 1788595051.129628 mapwrite 0x03f9e5 thru 0x03ffff (0x61b bytes) 000135 1788595051.222073 read 0x000d00 thru 0x0073b5 (0x66b6 bytes) 000136 1788595051.223204 write 0x0030d0 thru 0x00f824 (0xc755 bytes) 000137 1788595051.226077 mapwrite 0x03f669 thru 0x03ffff (0x997 bytes) 000138 1788595051.283902 write 0x005927 thru 0x01532b (0xfa05 bytes) 000139 1788595051.285271 mapwrite 0x012ac0 thru 0x01b547 (0x8a88 bytes) 000140 1788595051.349942 trunc from 0x040000 to 0x036fea (0x9015 bytes) 000141 1788595051.405481 punch from 0x02bd49 to 0x02c397 (0x64f bytes) 000142 1788595051.416723 mapwrite 0x03d3a3 thru 0x03ffff (0x2c5d bytes) 000143 1788595051.515418 read 0x011cf3 thru 0x0170fa (0x5408 bytes) 000144 1788595051.517441 mapread 0x03736c thru 0x03db55 (0x67ea bytes) 000146 1788595051.529506 mapwrite 0x004501 thru 0x0133df (0xeedf bytes) 000148 1788595051.591688 read 0x0153f9 thru 0x0226b4 (0xd2bc bytes) 000149 1788595051.592325 mapwrite 0x01180a thru 0x01c11e (0xa915 bytes) 000150 1788595051.647379 punch from 0x00800f to 0x008319 (0x30b bytes) 000151 1788595051.653071 punch from 0x008ceb to 0x012cc6 (0x9fdc bytes) 000152 1788595051.658079 mapwrite 0x00b41d thru 0x0158a7 (0xa48b bytes) 000153 1788595051.714198 mapread 0x01798e thru 0x01b22a (0x389d bytes) 000156 1788595051.717321 read 0x0373dd thru 0x03ffff (0x8c23 bytes) 000159 1788595051.727540 mapread 0x01cfdb thru 0x01d976 (0x99c bytes) 000161 1788595051.737655 MIRROR_EXTEND to 2 mirrors 000162 1788595052.681912 punch from 0x039744 to 0x03ae8f (0x174c bytes) 000164 1788595052.750165 punch from 0x02cba1 to 0x03b5a4 (0xea04 bytes) 000166 1788595052.761570 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ad:0x0]: Invalid argument (22) 000169 1788595052.894075 MIRROR_EXTEND to 2 mirrors 000170 1788595053.824992 mapread 0x00e762 thru 0x01dcbe (0xf55d bytes) 000171 1788595053.877874 trunc from 0x040000 to 0x017631 (0x289ce bytes) 000172 1788595053.969356 mapread 0x00e458 thru 0x00e763 (0x30c bytes) 000173 1788595053.986091 read 0x0071c5 thru 0x014a9a (0xd8d6 bytes) 000174 1788595053.986766 punch from 0x0061d1 to 0x007beb (0x1a1b bytes) 000175 1788595053.994023 mapwrite 0x038ce4 thru 0x03f27b (0x6598 bytes) 000177 1788595054.063115 write 0x01a6c1 thru 0x02254e (0x7e8e bytes) 000179 1788595054.075218 mapwrite 0x03b956 thru 0x03ffff (0x46aa bytes) 000180 1788595054.135059 read 0x03b5fd thru 0x03b76c (0x170 bytes) 000181 1788595054.135633 write 0x022a29 thru 0x0251a5 (0x277d bytes) 000183 1788595054.143926 trunc from 0x040000 to 0x0052ee (0x3ad11 bytes) 000184 1788595054.182112 punch from 0x00125d to 0x0052ee (0x4092 bytes) 000185 1788595054.190222 write 0x032cb7 thru 0x0370de (0x4428 bytes) 000186 1788595054.193209 write 0x03f2a1 thru 0x03ffff (0xd5f bytes) 000187 1788595054.206956 MIRROR_RESYNC 2 mirrors 000188 1788595054.948948 MIRROR_RESYNC 2 mirrors 000189 1788595055.253657 punch from 0x015242 to 0x025171 (0xff30 bytes) 000190 1788595055.335320 write 0x026057 thru 0x02bc02 (0x5bac bytes) 000191 1788595055.345923 mapwrite 0x03e3e0 thru 0x03ffff (0x1c20 bytes) 000192 1788595055.419230 mapread 0x00faa0 thru 0x010adb (0x103c bytes) 000193 1788595055.433850 write 0x02831a thru 0x02b61e (0x3305 bytes) 000194 1788595055.434895 read 0x0163f3 thru 0x01d8f7 (0x7505 bytes) 000195 1788595055.441834 write 0x0384c0 thru 0x03a0a3 (0x1be4 bytes) 000196 1788595055.452149 read 0x039695 thru 0x03ffff (0x696b bytes) 000197 1788595055.487826 punch from 0x002b4b to 0x00a803 (0x7cb9 bytes) 000198 1788595055.520636 write 0x02d77d thru 0x03b113 (0xd997 bytes) 000199 1788595055.526281 trunc from 0x040000 to 0x001074 (0x3ef8b bytes) 000200 1788595055.581578 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1075): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1075): Mirror 2: 0xb6ee3fc1 (stale) CRC-32 checksum value for chunk [0, 0x1075): Mirror 1: 0xb6ee3fc1 Mirror 3: 0xb6ee3fc1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1075: skipped 000201 1788595056.617629 write 0x006121 thru 0x00cf43 (0x6e23 bytes) 000202 1788595056.693582 trunc from 0x00cf44 to 0x026425 (0x194e2 bytes) 000203 1788595056.737721 write 0x014911 thru 0x01e6c4 (0x9db4 bytes) 000204 1788595056.748111 mapwrite 0x036a05 thru 0x03f506 (0x8b02 bytes) 000206 1788595056.867977 read 0x029055 thru 0x02c445 (0x33f1 bytes) 000207 1788595056.874916 write 0x019b72 thru 0x027924 (0xddb3 bytes) 000208 1788595056.880236 punch from 0x02556f to 0x030d0e (0xb7a0 bytes) 000210 1788595056.892803 punch from 0x0291f0 to 0x035c7b (0xca8c bytes) 000212 1788595056.901505 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3f507): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f507): Mirror 2: 0xe66728f7 (stale) Mirror 3: 0xc80ecf11 (stale) CRC-32 checksum value for chunk [0, 0x3f507): Mirror 1: 0x3fde74da Mirror 4: 0x3fde74da lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f507: skipped 000214 1788595057.974125 write 0x006c9e thru 0x0087ce (0x1b31 bytes) 000216 1788595058.107986 trunc from 0x03f507 to 0x020cab (0x1e85b bytes) 000218 1788595058.138353 punch from 0x0202e6 to 0x020cab (0x9c6 bytes) 000219 1788595058.151839 mapread 0x01a935 thru 0x020cab (0x6377 bytes) 000223 1788595058.182974 read 0x01bc25 thru 0x020cab (0x5087 bytes) 000224 1788595058.184081 mapread 0x0194fe thru 0x020cab (0x77ae bytes) 000225 1788595058.186078 mapwrite 0x039b9a thru 0x03f863 (0x5cca bytes) 000226 1788595058.284918 write 0x034c99 thru 0x036a9a (0x1e02 bytes) 000229 1788595058.286861 mapread 0x01a8fb thru 0x02423d (0x9943 bytes) 000230 1788595058.315204 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f864): Mirror 3: 0x4bd8d883 (stale) Mirror 4: 0x9d543197 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f864): Mirror 3: 0x4bd8d883 (stale) Mirror 4: 0x9d543197 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f864: skipped 000231 1788595058.688091 mapread 0x0181a1 thru 0x0243bc (0xc21c bytes) 000232 1788595058.728160 MIRROR_RESYNC 3 mirrors 000233 1788595059.952656 trunc from 0x03f864 to 0x02904b (0x16818 bytes) 000235 1788595060.028781 mapwrite 0x00757a thru 0x0140b3 (0xcb3a bytes) 000237 1788595060.154726 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2904c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2904c): Mirror 3: 0xac0ca747 (stale) Mirror 4: 0xac0ca747 (stale) CRC-32 checksum value for chunk [0, 0x2904c): Mirror 1: 0xcd07b47a Mirror 5: 0xcd07b47a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2904c: skipped 000238 1788595061.127426 mapread 0x0098f6 thru 0x015463 (0xbb6e bytes) 000239 1788595061.182569 mapread 0x005f98 thru 0x0073f7 (0x1460 bytes) 000240 1788595061.211020 punch from 0x00fd37 to 0x01cc20 (0xceea bytes) 000241 1788595061.311377 read 0x0114a1 thru 0x014de4 (0x3944 bytes) 000242 1788595061.320944 write 0x031e59 thru 0x03fd4c (0xdef4 bytes) 000244 1788595061.330581 read 0x00a1cf thru 0x015f13 (0xbd45 bytes) 000245 1788595061.358311 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3fd4d): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fd4d): Mirror 3: 0xc449a32d (stale) Mirror 4: 0xc449a32d (stale) Mirror 5: 0xbf6081fb (stale) CRC-32 checksum value for chunk [0, 0x3fd4d): Mirror 1: 0x8723d656 Mirror 6: 0x8723d656 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fd4d: skipped 000246 1788595062.874167 write 0x008a03 thru 0x0174d3 (0xead1 bytes) 000249 1788595062.960099 mapwrite 0x00681d thru 0x00f904 (0x90e8 bytes) 000255 1788595063.022998 read 0x005fa0 thru 0x00ce19 (0x6e7a bytes) 000257 1788595063.029010 mapread 0x03e46a thru 0x03fd4c (0x18e3 bytes) 000258 1788595063.038273 punch from 0x0252d6 to 0x033557 (0xe282 bytes) 000260 1788595063.047008 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fd4d): Mirror 3: 0xc449a32d (stale) Mirror 4: 0xc449a32d (stale) Mirror 5: 0xbf6081fb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fd4d): Mirror 3: 0xc449a32d (stale) Mirror 4: 0xc449a32d (stale) Mirror 5: 0xbf6081fb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fd4d: skipped 000261 1788595063.385162 write 0x02f5af thru 0x03c25f (0xccb1 bytes) 000262 1788595063.395893 read 0x00d9e8 thru 0x01d5d2 (0xfbeb bytes) 000265 1788595063.404037 mapwrite 0x036f9a thru 0x03ef43 (0x7faa bytes) 000266 1788595063.477766 punch from 0x0300a4 to 0x03aada (0xaa37 bytes) 000267 1788595063.483545 write 0x02dd0d thru 0x036b4c (0x8e40 bytes) 000271 1788595063.497048 read 0x02d2e0 thru 0x034dc8 (0x7ae9 bytes) 000274 1788595063.499292 read 0x008143 thru 0x00dd0f (0x5bcd bytes) 000275 1788595063.508994 trunc from 0x03fd4d to 0x00f7d1 (0x3057b bytes) 000277 1788595063.523057 write 0x032aa3 thru 0x032d80 (0x2de bytes) 000280 1788595063.525075 mapwrite 0x03b60a thru 0x03ffff (0x49f6 bytes) 000281 1788595063.597181 mapread 0x029e39 thru 0x03702c (0xd1f4 bytes) 000282 1788595063.653222 punch from 0x015a84 to 0x01ab5e (0x50db bytes) 000283 1788595063.657208 read 0x034e5e thru 0x03b3ff (0x65a2 bytes) 000284 1788595063.670031 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xb2d25399 (stale) Mirror 4: 0xb2d25399 (stale) Mirror 5: 0x276ba9f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4876494c Mirror 6: 0x4876494c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000287 1788595065.176910 mapread 0x02451d thru 0x03259a (0xe07e bytes) 000288 1788595065.203825 punch from 0x018b93 to 0x01a0f5 (0x1563 bytes) 000290 1788595065.288410 trunc from 0x040000 to 0x02d383 (0x12c7c bytes) 000291 1788595065.307743 read 0x000991 thru 0x00442e (0x3a9e bytes) 000292 1788595065.318789 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2d384): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d384): Mirror 3: 0xa4d39825 (stale) Mirror 4: 0xa4d39825 (stale) Mirror 5: 0xe317167d (stale) Mirror 6: 0xbc1445a (stale) CRC-32 checksum value for chunk [0, 0x2d384): Mirror 1: 0xbc1445a Mirror 7: 0xbc1445a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d384: skipped 000293 1788595066.620915 punch from 0x023246 to 0x02b566 (0x8321 bytes) 000294 1788595066.727034 mapread 0x0295d2 thru 0x02d383 (0x3db2 bytes) 000295 1788595066.763697 punch from 0x00e309 to 0x00f35d (0x1055 bytes) 000296 1788595066.782116 MIRROR_RESYNC 6 mirrors 000297 1788595068.874005 mapwrite 0x036e3f thru 0x03ffff (0x91c1 bytes) 000298 1788595069.006716 punch from 0x007c5e to 0x013a94 (0xbe37 bytes) 000299 1788595069.012793 write 0x008d9e thru 0x00cb2d (0x3d90 bytes) 000300 1788595069.023563 write 0x0174cd thru 0x01beba (0x49ee bytes) 000301 1788595069.034724 read 0x027ebd thru 0x02e416 (0x655a bytes) 000303 1788595069.047240 punch from 0x02898f to 0x02cf41 (0x45b3 bytes) 000304 1788595069.060919 mapread 0x002854 thru 0x004ee1 (0x268e bytes) 000305 1788595069.072183 mapread 0x00e1dc thru 0x0190f8 (0xaf1d bytes) 000306 1788595069.082951 mapwrite 0x02d14c thru 0x031c3f (0x4af4 bytes) 000307 1788595069.161962 MIRROR_RESYNC 6 mirrors 000308 1788595072.329109 write 0x0118ff thru 0x01d83f (0xbf41 bytes) 000309 1788595072.395522 mapread 0x013ab8 thru 0x013c85 (0x1ce bytes) 000311 1788595072.396868 trunc from 0x040000 to 0x0276e2 (0x1891d bytes) 000313 1788595072.428124 mapwrite 0x038c57 thru 0x03a64b (0x19f5 bytes) 000314 1788595072.549817 mapread 0x02b1c4 thru 0x02c5cd (0x140a bytes) 000315 1788595072.561338 MIRROR_RESYNC 6 mirrors 000316 1788595075.120914 read 0x0285eb thru 0x032291 (0x9ca7 bytes) 000317 1788595075.145904 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ad:0x0]: Invalid argument (22) 000318 1788595075.425515 write 0x027f5c thru 0x0328ec (0xa991 bytes) 000320 1788595075.512313 mapread 0x037318 thru 0x03a64b (0x3334 bytes) 000322 1788595075.528689 trunc from 0x03a64c to 0x00d700 (0x2cf4b bytes) 000323 1788595075.548814 read 0x008391 thru 0x00b403 (0x3073 bytes) 000324 1788595075.562461 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xd701): Mirror 3: 0xb6d26aa4 (stale) Mirror 6: 0xb6d26aa4 (stale) Mirror 7: 0xb6d26aa4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd701): Mirror 3: 0xb6d26aa4 (stale) Mirror 6: 0xb6d26aa4 (stale) Mirror 7: 0xb6d26aa4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd701: skipped 000325 1788595076.038623 trunc from 0x00d701 to 0x013329 (0x5c29 bytes) 000326 1788595076.055519 read 0x00f684 thru 0x012c88 (0x3605 bytes) 000328 1788595076.062011 read 0x009a6c thru 0x01168f (0x7c24 bytes) 000329 1788595076.070418 punch from 0x009c36 to 0x013329 (0x96f4 bytes) 000332 1788595076.075957 mapread 0x001d68 thru 0x0068b3 (0x4b4c bytes) 000333 1788595076.080705 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1332a): Mirror 6: 0xcd59938c (stale) Mirror 7: 0xcd59938c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1332a): Mirror 6: 0xcd59938c (stale) Mirror 7: 0xcd59938c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1332a: skipped 000334 1788595076.406566 mapread 0x011fd2 thru 0x013329 (0x1358 bytes) 000336 1788595076.425983 mapread 0x003477 thru 0x009e62 (0x69ec bytes) 000337 1788595076.435103 write 0x03fee3 thru 0x03ffff (0x11d bytes) 000340 1788595076.438207 read 0x007e27 thru 0x01143a (0x9614 bytes) 000341 1788595076.439122 read 0x0009e7 thru 0x0010ef (0x709 bytes) 000342 1788595076.452118 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xa7e3145e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xa7e3145e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000343 1788595076.781150 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ad:0x0]: Invalid argument (22) 000344 1788595076.968145 mapwrite 0x02e4ba thru 0x03b9b2 (0xd4f9 bytes) 000345 1788595077.094773 write 0x00c307 thru 0x01b606 (0xf300 bytes) 000347 1788595077.116793 punch from 0x01f754 to 0x0267c5 (0x7072 bytes) truncating to largest ever: 0x3cf85 000348 1788595077.130044 trunc from 0x040000 to 0x03cf84 (0x307b bytes) 000349 1788595077.166194 punch from 0x01d5ad to 0x023419 (0x5e6d bytes) 000350 1788595077.173599 read 0x018a07 thru 0x023a45 (0xb03f bytes) 000351 1788595077.186790 trunc from 0x03cf85 to 0x009c5a (0x3332a bytes) 000352 1788595077.216265 punch from 0x0020d4 to 0x009c5a (0x7b87 bytes) 000355 1788595077.231055 trunc from 0x009c5b to 0x021a8f (0x17e35 bytes) 000357 1788595077.246218 mapread 0x001ef4 thru 0x002add (0xbea bytes) 000358 1788595077.258940 read 0x006505 thru 0x00f363 (0x8e5f bytes) 000359 1788595077.260027 read 0x01e9d4 thru 0x021082 (0x26af bytes) 000360 1788595077.277349 write 0x0174b8 thru 0x017634 (0x17d bytes) 000362 1788595077.278654 trunc from 0x021a90 to 0x039359 (0x178ca bytes) 000363 1788595077.319863 write 0x0258d8 thru 0x027030 (0x1759 bytes) 000364 1788595077.340763 write 0x0339df thru 0x03e906 (0xaf28 bytes) 000366 1788595077.342386 read 0x03c816 thru 0x03e906 (0x20f1 bytes) 000368 1788595077.353851 write 0x01515f thru 0x020103 (0xafa5 bytes) 000370 1788595077.356589 write 0x03d984 thru 0x03ffff (0x267c bytes) 000372 1788595077.375554 MIRROR_EXTEND to 2 mirrors 000373 1788595078.352006 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ad:0x0]: Invalid argument (22) 000375 1788595078.531175 write 0x038411 thru 0x03f60f (0x71ff bytes) 000376 1788595078.552984 punch from 0x03a54f to 0x03bdef (0x18a1 bytes) 000379 1788595078.576560 trunc from 0x040000 to 0x03109c (0xef63 bytes) 000380 1788595078.615472 read 0x01d932 thru 0x01e0f3 (0x7c2 bytes) 000381 1788595078.639008 mapwrite 0x01099f thru 0x011917 (0xf79 bytes) 000382 1788595078.777795 read 0x009582 thru 0x010f6d (0x79ec bytes) 000383 1788595078.793070 read 0x00169a thru 0x0067ae (0x5115 bytes) 000385 1788595078.794260 read 0x008d4a thru 0x00c269 (0x3520 bytes) 000386 1788595078.794996 punch from 0x024729 to 0x02b062 (0x693a bytes) 000387 1788595078.801114 mapwrite 0x01d513 thru 0x02b112 (0xdc00 bytes) 000390 1788595078.855724 mapread 0x02025c thru 0x023711 (0x34b6 bytes) 000391 1788595078.856742 read 0x01f49b thru 0x02f1bf (0xfd25 bytes) 000392 1788595078.857810 punch from 0x026371 to 0x0267f3 (0x483 bytes) 000393 1788595078.861529 read 0x030451 thru 0x03109c (0xc4c bytes) 000394 1788595078.862903 mapread 0x01764e thru 0x018cc9 (0x167c bytes) 000395 1788595078.863937 trunc from 0x03109d to 0x00a0c7 (0x26fd5 bytes) 000398 1788595078.879964 write 0x01869f thru 0x0237fb (0xb15d bytes) 000399 1788595078.881852 trunc from 0x0237fc to 0x00ca7a (0x16d81 bytes) 000400 1788595078.895525 write 0x01dff5 thru 0x02aba3 (0xcbaf bytes) 000402 1788595078.898518 punch from 0x01007e to 0x017149 (0x70cc bytes) 000403 1788595078.911145 punch from 0x00e83d to 0x014211 (0x59d5 bytes) 000405 1788595078.915817 mapread 0x017ea6 thru 0x027ca5 (0xfe00 bytes) 000406 1788595078.923737 mapwrite 0x0270ef thru 0x0291c0 (0x20d2 bytes) 000409 1788595078.968337 mapread 0x0069d1 thru 0x009142 (0x2772 bytes) 000410 1788595078.969159 trunc from 0x02aba4 to 0x026cdc (0x3ec7 bytes) 000412 1788595078.981080 trunc from 0x026cdd to 0x01abf0 (0xc0ec bytes) 000414 1788595078.996235 write 0x009ae8 thru 0x00b9ca (0x1ee3 bytes) 000416 1788595078.997734 punch from 0x010393 to 0x0185a3 (0x8211 bytes) 000417 1788595079.010216 mapread 0x00a8ab thru 0x0184e7 (0xdc3d bytes) 000418 1788595079.075058 trunc from 0x01abf1 to 0x00b913 (0xf2dd bytes) 000419 1788595079.103746 punch from 0x006594 to 0x007c24 (0x1691 bytes) 000420 1788595079.109742 MIRROR_EXTEND to 2 mirrors 000421 1788595079.833525 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ad:0x0]: Invalid argument (22) 000422 1788595080.058597 MIRROR_EXTEND to 2 mirrors 000423 1788595080.873620 punch from 0x007a88 to 0x00b913 (0x3e8c bytes) 000424 1788595080.948095 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xb914): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb914): Mirror 2: 0x83ba7fef (stale) CRC-32 checksum value for chunk [0, 0xb914): Mirror 1: 0x62a5c0e2 Mirror 3: 0x62a5c0e2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb914: skipped 000425 1788595081.739004 mapread 0x003e6b thru 0x0052b7 (0x144d bytes) 000426 1788595081.771407 mapread 0x008368 thru 0x00b913 (0x35ac bytes) 000427 1788595081.776367 read 0x00b8e4 thru 0x00b913 (0x30 bytes) 000428 1788595081.776871 punch from 0x0012ea to 0x00a030 (0x8d47 bytes) 000429 1788595081.906974 mapread 0x006add thru 0x00b913 (0x4e37 bytes) 000430 1788595081.915147 read 0x0044be thru 0x00a5f4 (0x6137 bytes) 000431 1788595081.918720 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xb914): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb914): Mirror 2: 0x83ba7fef (stale) Mirror 3: 0x62a5c0e2 (stale) CRC-32 checksum value for chunk [0, 0xb914): Mirror 1: 0x62a5c0e2 Mirror 4: 0x62a5c0e2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb914: skipped 000432 1788595082.934956 mapread 0x00218c thru 0x00b913 (0x9788 bytes) 000433 1788595083.003279 trunc from 0x00b914 to 0x0121f6 (0x68e3 bytes) 000434 1788595083.145794 mapread 0x001111 thru 0x007f6e (0x6e5e bytes) 000435 1788595083.161130 read 0x006c7a thru 0x011a42 (0xadc9 bytes) 000438 1788595083.175868 mapread 0x0119a6 thru 0x0121f6 (0x851 bytes) 000439 1788595083.191747 mapwrite 0x00ddde thru 0x00df25 (0x148 bytes) 000441 1788595083.347112 trunc from 0x0121f7 to 0x00018b (0x1206b bytes) 000442 1788595083.397051 trunc from 0x00018c to 0x017539 (0x173ae bytes) 000444 1788595083.423595 trunc from 0x01753a to 0x029d10 (0x127d7 bytes) 000446 1788595083.457783 punch from 0x019094 to 0x024904 (0xb871 bytes) 000448 1788595083.462972 read 0x015aee thru 0x01b906 (0x5e19 bytes) 000450 1788595083.476236 mapread 0x0270ea thru 0x029d10 (0x2c27 bytes) 000452 1788595083.483302 read 0x029908 thru 0x029d10 (0x409 bytes) 000453 1788595083.484436 mapwrite 0x01e682 thru 0x029cfc (0xb67b bytes) 000457 1788595083.612437 write 0x01c00e thru 0x026df2 (0xade5 bytes) 000460 1788595083.619266 read 0x0070fc thru 0x00a696 (0x359b bytes) 000462 1788595083.621692 mapwrite 0x0052f7 thru 0x00cd7b (0x7a85 bytes) 000463 1788595083.730850 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x29d11): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29d11): Mirror 2: 0x7cedf740 (stale) Mirror 3: 0x5bfce6d5 (stale) Mirror 4: 0x5bfce6d5 (stale) CRC-32 checksum value for chunk [0, 0x29d11): Mirror 1: 0x728aa0fa Mirror 5: 0x728aa0fa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29d11: skipped 000465 1788595084.761053 write 0x0006b2 thru 0x001593 (0xee2 bytes) 000467 1788595084.887949 mapwrite 0x01121c thru 0x0143f1 (0x31d6 bytes) 000468 1788595085.028226 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x29d11): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29d11): Mirror 2: 0x7cedf740 (stale) Mirror 3: 0x5bfce6d5 (stale) Mirror 4: 0x5bfce6d5 (stale) Mirror 5: 0x728aa0fa (stale) CRC-32 checksum value for chunk [0, 0x29d11): Mirror 1: 0x711bb148 Mirror 6: 0x711bb148 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29d11: skipped 000469 1788595086.447997 read 0x028460 thru 0x029d10 (0x18b1 bytes) 000470 1788595086.469483 MIRROR_RESYNC 6 mirrors 000471 1788595088.707424 punch from 0x020f82 to 0x029d10 (0x8d8f bytes) 000473 1788595088.782301 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29d11): Mirror 3: 0x711bb148 (stale) Mirror 4: 0x711bb148 (stale) Mirror 5: 0x711bb148 (stale) Mirror 6: 0x711bb148 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29d11): Mirror 3: 0x711bb148 (stale) Mirror 4: 0x711bb148 (stale) Mirror 5: 0x711bb148 (stale) Mirror 6: 0x711bb148 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29d11: skipped 000474 1788595089.173457 write 0x012a69 thru 0x017b9a (0x5132 bytes) 000475 1788595089.186113 trunc from 0x029d11 to 0x000c29 (0x290e7 bytes) 000478 1788595089.207387 write 0x03e4b7 thru 0x03ffff (0x1b49 bytes) 000481 1788595089.210610 mapwrite 0x02f1d2 thru 0x03af55 (0xbd84 bytes) 000482 1788595089.301639 read 0x02a9c2 thru 0x02ac74 (0x2b3 bytes) 000483 1788595089.321820 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x68eb7a22 (stale) Mirror 4: 0x68eb7a22 (stale) Mirror 5: 0x68eb7a22 (stale) Mirror 6: 0x68eb7a22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe0edd0eb Mirror 7: 0xe0edd0eb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000484 1788595090.725654 write 0x030e96 thru 0x0333f2 (0x255d bytes) 000485 1788595090.884965 punch from 0x0373d5 to 0x039779 (0x23a5 bytes) 000486 1788595090.892546 trunc from 0x040000 to 0x004629 (0x3b9d6 bytes) 000489 1788595090.994013 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x462a): Mirror 4: 0x1959c483 (stale) Mirror 5: 0x1959c483 (stale) Mirror 6: 0x1959c483 (stale) Mirror 7: 0x6e406fd0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x462a): Mirror 4: 0x1959c483 (stale) Mirror 5: 0x1959c483 (stale) Mirror 6: 0x1959c483 (stale) Mirror 7: 0x6e406fd0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x462a: skipped 000490 1788595091.545986 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x462a): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x462a): Mirror 4: 0x1959c483 (stale) Mirror 5: 0x1959c483 (stale) Mirror 6: 0x1959c483 (stale) Mirror 7: 0x6e406fd0 (stale) CRC-32 checksum value for chunk [0, 0x462a): Mirror 1: 0x6e406fd0 Mirror 8: 0x6e406fd0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x462a: skipped 000492 1788595092.752212 read 0x000b8a thru 0x004629 (0x3aa0 bytes) 000493 1788595092.771170 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x462a): Mirror 4: 0x1959c483 (stale) Mirror 5: 0x1959c483 (stale) Mirror 7: 0x6e406fd0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x462a): Mirror 4: 0x1959c483 (stale) Mirror 5: 0x1959c483 (stale) Mirror 7: 0x6e406fd0 (stale) CRC-32 checksum value for chunk [0, 0x462a): Mirror 1: 0x6e406fd0 Mirror 8: 0x6e406fd0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x462a: skipped 000494 1788595093.378031 trunc from 0x00462a to 0x017302 (0x12cd9 bytes) 000495 1788595093.498086 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x17303): Mirror 5: 0xea8f6c4e (stale) Mirror 7: 0xc823a37e (stale) Mirror 8: 0xc823a37e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17303): Mirror 5: 0xea8f6c4e (stale) Mirror 7: 0xc823a37e (stale) Mirror 8: 0xc823a37e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17303: skipped 000496 1788595093.844475 mapread 0x006ce0 thru 0x00a257 (0x3578 bytes) 000497 1788595093.864543 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x17303): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17303): Mirror 5: 0xea8f6c4e (stale) Mirror 7: 0xc823a37e (stale) Mirror 8: 0xc823a37e (stale) CRC-32 checksum value for chunk [0, 0x17303): Mirror 1: 0xc823a37e Mirror 9: 0xc823a37e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17303: skipped 000498 1788595094.910997 mapwrite 0x02dd88 thru 0x037c51 (0x9eca bytes) 000499 1788595095.154606 read 0x02b3f9 thru 0x036c62 (0xb86a bytes) 000500 1788595095.156413 write 0x024bd4 thru 0x02c203 (0x7630 bytes) 000501 1788595095.175054 read 0x0354c4 thru 0x037c51 (0x278e bytes) 000502 1788595095.177020 mapread 0x02f589 thru 0x037c51 (0x86c9 bytes) 000503 1788595095.179392 write 0x03e500 thru 0x03ffff (0x1b00 bytes) 000504 1788595095.187959 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xe0edd0eb (stale) Mirror 8: 0xc4d441a3 (stale) Mirror 9: 0xc4d441a3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xe0edd0eb (stale) Mirror 8: 0xc4d441a3 (stale) Mirror 9: 0xc4d441a3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000505 1788595095.753575 mapread 0x03b5ad thru 0x03ffff (0x4a53 bytes) 000506 1788595095.786790 write 0x03372a thru 0x03ffff (0xc8d6 bytes) 000507 1788595095.795454 write 0x03f116 thru 0x03ffff (0xeea bytes) 000508 1788595095.799552 MIRROR_RESYNC 4 mirrors 000511 1788595097.284596 mapwrite 0x035915 thru 0x03a593 (0x4c7f bytes) 000516 1788595097.498617 mapwrite 0x00c4c6 thru 0x00fcf1 (0x382c bytes) 000518 1788595097.582623 mapread 0x034b93 thru 0x03ffff (0xb46d bytes) 000519 1788595097.593307 trunc from 0x040000 to 0x00feb9 (0x30146 bytes) 000521 1788595097.619062 write 0x01ec1f thru 0x01ff19 (0x12fb bytes) 000522 1788595097.628971 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1ff1a): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ff1a): Mirror 7: 0x88da97da (stale) Mirror 8: 0x88da97da (stale) Mirror 9: 0x88da97da (stale) CRC-32 checksum value for chunk [0, 0x1ff1a): Mirror 1: 0xd4b32cbe Mirror 10: 0xd4b32cbe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ff1a: skipped 000523 1788595098.931212 trunc from 0x01ff1a to 0x035fd8 (0x160bf bytes) 000524 1788595099.014046 write 0x0009b6 thru 0x01089d (0xfee8 bytes) 000525 1788595099.029468 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x35fd9): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35fd9): Mirror 7: 0x45cf0326 (stale) Mirror 8: 0x45cf0326 (stale) Mirror 9: 0x45cf0326 (stale) Mirror 10: 0x6332af9e (stale) CRC-32 checksum value for chunk [0, 0x35fd9): Mirror 1: 0x3ec809a6 Mirror 11: 0x3ec809a6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35fd9: skipped 000526 1788595100.058142 punch from 0x007ea1 to 0x0130cc (0xb22c bytes) 000527 1788595100.133532 mapwrite 0x01d6dc thru 0x01dc8d (0x5b2 bytes) 000530 1788595100.180000 trunc from 0x035fd9 to 0x036bd3 (0xbfb bytes) 000531 1788595100.204563 mapread 0x008fad thru 0x00c7c9 (0x381d bytes) 000532 1788595100.219520 read 0x008ffb thru 0x01454f (0xb555 bytes) 000533 1788595100.227701 read 0x031b21 thru 0x036bd3 (0x50b3 bytes) 000535 1788595100.253870 punch from 0x017d9f to 0x01ff3b (0x819d bytes) 000536 1788595100.262480 read 0x02b1e2 thru 0x034a32 (0x9851 bytes) 000537 1788595100.266986 read 0x00e02c thru 0x0163be (0x8393 bytes) 000541 1788595100.274739 mapwrite 0x02439f thru 0x02aff6 (0x6c58 bytes) 000542 1788595100.361876 mapread 0x01cc34 thru 0x02a559 (0xd926 bytes) 000543 1788595100.370430 write 0x01e346 thru 0x023084 (0x4d3f bytes) 000544 1788595100.371442 write 0x03c2fc thru 0x03ffff (0x3d04 bytes) 000545 1788595100.372356 mapwrite 0x01b9c9 thru 0x02a6a9 (0xece1 bytes) 000546 1788595100.432857 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x3d98c744 (stale) Mirror 8: 0x3d98c744 (stale) Mirror 9: 0x3d98c744 (stale) Mirror 10: 0xfff9c3a4 (stale) Mirror 11: 0x97fdd7cf (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbee57fae Mirror 12: 0xbee57fae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000548 1788595101.852010 MIRROR_RESYNC 7 mirrors 000549 1788595104.125576 trunc from 0x040000 to 0x015739 (0x2a8c6 bytes) 000550 1788595104.207098 mapread 0x0091e5 thru 0x015739 (0xc555 bytes) 000551 1788595104.230575 mapread 0x00c1d9 thru 0x00eca2 (0x2aca bytes) 000553 1788595104.232874 read 0x005053 thru 0x006a21 (0x19cf bytes) 000555 1788595104.234422 mapwrite 0x00e0bc thru 0x016d90 (0x8cd5 bytes) 000556 1788595104.332924 trunc from 0x016d91 to 0x01e9ce (0x7c3e bytes) truncating to largest ever: 0x3da2d 000557 1788595104.354357 trunc from 0x01e9cf to 0x03da2c (0x1f05e bytes) 000558 1788595104.381578 trunc from 0x03da2d to 0x0181a0 (0x2588c bytes) 000559 1788595104.401250 mapread 0x00f0fb thru 0x0181a0 (0x90a6 bytes) 000560 1788595104.406534 mapread 0x015097 thru 0x0181a0 (0x310a bytes) 000562 1788595104.408543 mapwrite 0x01c28f thru 0x022f23 (0x6c95 bytes) 000564 1788595104.499656 write 0x00dffc thru 0x00f75e (0x1763 bytes) 000567 1788595104.502750 read 0x007405 thru 0x00c614 (0x5210 bytes) 000568 1788595104.503354 read 0x01fde9 thru 0x022f23 (0x313b bytes) 000569 1788595104.503965 write 0x02e1e4 thru 0x03bd16 (0xdb33 bytes) 000570 1788595104.509908 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bd17): Mirror 7: 0x5a015c3f (stale) Mirror 8: 0x5a015c3f (stale) Mirror 9: 0x5a015c3f (stale) Mirror 10: 0x5a015c3f (stale) Mirror 11: 0x5a015c3f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bd17): Mirror 7: 0x5a015c3f (stale) Mirror 8: 0x5a015c3f (stale) Mirror 9: 0x5a015c3f (stale) Mirror 10: 0x5a015c3f (stale) Mirror 11: 0x5a015c3f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bd17: skipped 000572 1788595104.897620 punch from 0x018cd2 to 0x025c13 (0xcf42 bytes) 000574 1788595104.902784 read 0x026c49 thru 0x02a2f8 (0x36b0 bytes) 000575 1788595104.916462 mapread 0x02c13a thru 0x031cda (0x5ba1 bytes) 000576 1788595104.943297 trunc from 0x03bd17 to 0x00730e (0x34a08 bytes) 000577 1788595104.962927 trunc from 0x00730f to 0x011fc5 (0xacb7 bytes) 000578 1788595104.998879 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x11fc6): Mirror 8: 0x2b7bf48 (stale) Mirror 9: 0x2b7bf48 (stale) Mirror 10: 0x2b7bf48 (stale) Mirror 11: 0x2b7bf48 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11fc6): Mirror 8: 0x2b7bf48 (stale) Mirror 9: 0x2b7bf48 (stale) Mirror 10: 0x2b7bf48 (stale) Mirror 11: 0x2b7bf48 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11fc6: skipped 000581 1788595105.437315 trunc from 0x011fc6 to 0x002cf6 (0xf2cf bytes) 000582 1788595105.460959 write 0x00dbdb thru 0x0176cd (0x9af3 bytes) 000583 1788595105.463185 mapread 0x015efb thru 0x0176cd (0x17d3 bytes) 000584 1788595105.471778 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x176ce): Mirror 8: 0xed39d1b5 (stale) Mirror 9: 0xed39d1b5 (stale) Mirror 11: 0xed39d1b5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x176ce): Mirror 8: 0xed39d1b5 (stale) Mirror 9: 0xed39d1b5 (stale) Mirror 11: 0xed39d1b5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x176ce: skipped 000585 1788595105.865896 punch from 0x00ef3f to 0x0176cd (0x878f bytes) 000586 1788595105.875434 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x176ce): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x176ce): Mirror 8: 0xed39d1b5 (stale) Mirror 9: 0xed39d1b5 (stale) Mirror 11: 0xed39d1b5 (stale) CRC-32 checksum value for chunk [0, 0x176ce): Mirror 1: 0x3c17d491 Mirror 12: 0x3c17d491 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x176ce: skipped 000588 1788595106.924021 write 0x0324e7 thru 0x03e31e (0xbe38 bytes) 000590 1788595107.022059 mapwrite 0x023294 thru 0x0251e1 (0x1f4e bytes) 000593 1788595107.119016 mapwrite 0x00ce19 thru 0x0166be (0x98a6 bytes) 000594 1788595107.280273 mapwrite 0x01207f thru 0x018c1b (0x6b9d bytes) 000595 1788595107.384296 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3e31f): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e31f): Mirror 8: 0xa7bea45d (stale) Mirror 9: 0xa7bea45d (stale) Mirror 11: 0xa7bea45d (stale) Mirror 12: 0xb899df6c (stale) CRC-32 checksum value for chunk [0, 0x3e31f): Mirror 1: 0xc38fb263 Mirror 13: 0xc38fb263 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e31f: skipped 000596 1788595108.615544 write 0x0297ef thru 0x02b496 (0x1ca8 bytes) 000598 1788595108.713196 write 0x03bd62 thru 0x03f30a (0x35a9 bytes) 000599 1788595108.736170 read 0x01c781 thru 0x01e8f9 (0x2179 bytes) 000600 1788595108.752011 punch from 0x00658b to 0x0110d2 (0xab48 bytes) 000602 1788595108.772314 write 0x001985 thru 0x00e63f (0xccbb bytes) 000605 1788595108.792078 punch from 0x01aedf to 0x0276e6 (0xc808 bytes) 000606 1788595108.813343 mapwrite 0x023527 thru 0x02dffd (0xaad7 bytes) 000609 1788595108.875059 MIRROR_RESYNC 6 mirrors 000610 1788595111.072644 mapwrite 0x018a74 thru 0x01ee74 (0x6401 bytes) 000613 1788595111.211968 write 0x02bd12 thru 0x03227e (0x656d bytes) 000614 1788595111.221881 mapread 0x029db4 thru 0x0349c1 (0xac0e bytes) 000615 1788595111.237770 mapread 0x00049f thru 0x00937c (0x8ede bytes) 000616 1788595111.246730 trunc from 0x03f30b to 0x002b00 (0x3c80a bytes) 000617 1788595111.269424 punch from 0x001938 to 0x002b00 (0x11c9 bytes) 000618 1788595111.273852 punch from 0x000fed to 0x002b00 (0x1b14 bytes) 000619 1788595111.282849 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b01): Mirror 8: 0xd11a8d3 (stale) Mirror 9: 0xd11a8d3 (stale) Mirror 12: 0xd11a8d3 (stale) Mirror 13: 0xd11a8d3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b01): Mirror 8: 0xd11a8d3 (stale) Mirror 9: 0xd11a8d3 (stale) Mirror 12: 0xd11a8d3 (stale) Mirror 13: 0xd11a8d3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b01: skipped 000621 1788595111.695313 read 0x001c01 thru 0x002b00 (0xf00 bytes) 000624 1788595111.708584 read 0x001698 thru 0x002b00 (0x1469 bytes) 000625 1788595111.709268 punch from 0x0018ea to 0x002b00 (0x1217 bytes) 000626 1788595111.716885 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b01): Mirror 8: 0xd11a8d3 (stale) Mirror 9: 0xd11a8d3 (stale) Mirror 13: 0xd11a8d3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b01): Mirror 8: 0xd11a8d3 (stale) Mirror 9: 0xd11a8d3 (stale) Mirror 13: 0xd11a8d3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b01: skipped 000628 1788595112.034495 write 0x028cb3 thru 0x031f01 (0x924f bytes) 000629 1788595112.039713 mapwrite 0x013bde thru 0x01aaaa (0x6ecd bytes) 000631 1788595112.144890 read 0x019850 thru 0x01d47e (0x3c2f bytes) 000632 1788595112.145720 mapwrite 0x000e50 thru 0x00a4e3 (0x9694 bytes) 000633 1788595112.205127 read 0x0236c8 thru 0x02a330 (0x6c69 bytes) 000635 1788595112.214962 read 0x00cb78 thru 0x013260 (0x66e9 bytes) 000636 1788595112.215812 write 0x025d74 thru 0x03226f (0xc4fc bytes) 000637 1788595112.218545 mapwrite 0x02495c thru 0x026e1d (0x24c2 bytes) 000639 1788595112.314470 punch from 0x016de0 to 0x016f02 (0x123 bytes) 000640 1788595112.322360 mapread 0x02efec thru 0x03226f (0x3284 bytes) 000641 1788595112.323276 mapwrite 0x016145 thru 0x023e54 (0xdd10 bytes) 000645 1788595112.437133 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32270): Mirror 9: 0x4803015c (stale) Mirror 13: 0x4803015c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32270): Mirror 9: 0x4803015c (stale) Mirror 13: 0x4803015c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32270: skipped 000652 1788595112.769736 mapwrite 0x02ebce thru 0x037cdd (0x9110 bytes) 000653 1788595112.852488 write 0x001c0f thru 0x00b9c4 (0x9db6 bytes) 000654 1788595112.866037 write 0x03751f thru 0x03ffff (0x8ae1 bytes) 000659 1788595112.871272 trunc from 0x040000 to 0x032401 (0xdbfe bytes) 000660 1788595112.888600 mapread 0x007540 thru 0x0128d1 (0xb392 bytes) 000663 1788595112.897958 trunc from 0x032402 to 0x030b70 (0x1891 bytes) 000664 1788595112.913443 write 0x035418 thru 0x03ffff (0xabe8 bytes) 000665 1788595112.915438 mapread 0x03bc96 thru 0x03ffff (0x436a bytes) 000667 1788595112.917082 punch from 0x03625b to 0x03a46e (0x4214 bytes) 000668 1788595112.939438 mapwrite 0x037695 thru 0x03daa4 (0x6410 bytes) 000669 1788595113.017386 read 0x005e33 thru 0x014f1d (0xf0eb bytes) 000670 1788595113.022716 MIRROR_RESYNC 3 mirrors 000671 1788595114.139117 punch from 0x039604 to 0x03ffff (0x69fc bytes) 000673 1788595114.211542 trunc from 0x040000 to 0x013ecc (0x2c133 bytes) 000674 1788595114.252072 read 0x00a15c thru 0x00ffd7 (0x5e7c bytes) 000675 1788595114.273297 trunc from 0x013ecd to 0x02a49b (0x165cf bytes) 000676 1788595114.290805 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a49c): Mirror 13: 0x3f2b3687 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a49c): Mirror 13: 0x3f2b3687 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a49c: skipped 000678 1788595114.718806 write 0x0012cc thru 0x00dbc3 (0xc8f8 bytes) 000680 1788595114.729507 write 0x021141 thru 0x03033b (0xf1fb bytes) 000681 1788595114.734939 write 0x01eae5 thru 0x026bd4 (0x80f0 bytes) 000684 1788595114.743804 write 0x033777 thru 0x03cc36 (0x94c0 bytes) 000685 1788595114.745865 mapread 0x02de36 thru 0x0376f0 (0x98bb bytes) truncating to largest ever: 0x3e01e 000687 1788595114.753743 trunc from 0x03cc37 to 0x03e01d (0x13e7 bytes) 000689 1788595114.773176 mapwrite 0x02c304 thru 0x02dc07 (0x1904 bytes) 000690 1788595114.878297 mapwrite 0x0355b3 thru 0x03ffff (0xaa4d bytes) 000691 1788595114.994211 mapread 0x03d948 thru 0x03ffff (0x26b8 bytes) 000692 1788595114.996130 trunc from 0x040000 to 0x0356d8 (0xa927 bytes) 000693 1788595115.032637 punch from 0x010e67 to 0x01a344 (0x94de bytes) 000694 1788595115.038326 write 0x03850b thru 0x03ef74 (0x6a6a bytes) 000695 1788595115.041827 write 0x0302c7 thru 0x03b76f (0xb4a9 bytes) 000696 1788595115.045943 mapread 0x028010 thru 0x02b888 (0x3879 bytes) 000698 1788595115.047535 mapwrite 0x02f989 thru 0x03d167 (0xd7df bytes) 000699 1788595115.135595 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ad:0x0]: Invalid argument (22) 000702 1788595115.294412 read 0x034cf7 thru 0x03ef74 (0xa27e bytes) 000710 1788595115.311096 read 0x038403 thru 0x03ef74 (0x6b72 bytes) 000712 1788595115.312419 trunc from 0x03ef75 to 0x011a36 (0x2d53e bytes) 000713 1788595115.326181 trunc from 0x011a37 to 0x021689 (0xfc53 bytes) 000714 1788595115.345500 mapwrite 0x01ff02 thru 0x02d910 (0xda0f bytes) 000715 1788595115.421223 punch from 0x007b25 to 0x0110bb (0x9597 bytes) 000718 1788595115.429400 punch from 0x0055db to 0x014ef5 (0xf91b bytes) 000719 1788595115.433941 read 0x019b1d thru 0x02410d (0xa5f1 bytes) 000721 1788595115.439162 mapwrite 0x02cc62 thru 0x039e55 (0xd1f4 bytes) 000724 1788595115.512435 mapread 0x033d69 thru 0x039e55 (0x60ed bytes) 000725 1788595115.513623 mapwrite 0x03b5bb thru 0x03d039 (0x1a7f bytes) 000728 1788595115.579249 MIRROR_EXTEND to 2 mirrors 000730 1788595116.701957 punch from 0x00c409 to 0x00ff34 (0x3b2c bytes) 000731 1788595116.812426 read 0x004163 thru 0x0086c0 (0x455e bytes) 000732 1788595116.820657 mapread 0x01a9c4 thru 0x020437 (0x5a74 bytes) 000733 1788595116.851691 mapread 0x005622 thru 0x00ea6c (0x944b bytes) 000734 1788595116.860602 write 0x01640d thru 0x01d90c (0x7500 bytes) 000735 1788595116.863440 mapwrite 0x016114 thru 0x0237b2 (0xd69f bytes) 000738 1788595116.924867 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3d03a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d03a): Mirror 2: 0x720aa023 (stale) CRC-32 checksum value for chunk [0, 0x3d03a): Mirror 1: 0x516f19a7 Mirror 3: 0x516f19a7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d03a: skipped 000739 1788595117.911215 punch from 0x02239f to 0x025ab4 (0x3716 bytes) 000740 1788595118.101234 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d03a): Mirror 3: 0x516f19a7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d03a): Mirror 3: 0x516f19a7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d03a: skipped 000742 1788595118.477045 write 0x026602 thru 0x027af1 (0x14f0 bytes) 000743 1788595118.486004 mapread 0x009c1c thru 0x01583f (0xbc24 bytes) 000744 1788595118.517697 trunc from 0x03d03a to 0x012613 (0x2aa26 bytes) 000745 1788595118.542134 mapread 0x0012f2 thru 0x00e152 (0xce61 bytes) 000746 1788595118.554411 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x12614): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12614): Mirror 3: 0x61faa112 (stale) CRC-32 checksum value for chunk [0, 0x12614): Mirror 1: 0x61faa112 Mirror 4: 0x61faa112 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12614: skipped 000747 1788595119.575343 punch from 0x00b708 to 0x01019e (0x4a97 bytes) 000748 1788595119.670449 trunc from 0x012614 to 0x00a3ca (0x8249 bytes) 000754 1788595119.706822 MIRROR_RESYNC 3 mirrors 000755 1788595120.550162 write 0x01aea9 thru 0x028a1a (0xdb72 bytes) 000756 1788595120.668449 mapwrite 0x0001ae thru 0x008b88 (0x89db bytes) 000758 1788595120.785614 write 0x00b2ee thru 0x00b3bc (0xcf bytes) 000759 1788595120.793244 punch from 0x026ca7 to 0x028a1a (0x1d74 bytes) 000761 1788595120.810816 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28a1b): Mirror 4: 0x550ca377 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28a1b): Mirror 4: 0x550ca377 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28a1b: skipped 000762 1788595121.143333 mapwrite 0x0183ad thru 0x021d5f (0x99b3 bytes) 000763 1788595121.231040 punch from 0x01be92 to 0x025f94 (0xa103 bytes) 000764 1788595121.245326 write 0x01ccd9 thru 0x02568a (0x89b2 bytes) 000765 1788595121.272461 punch from 0x00711a to 0x015556 (0xe43d bytes) 000766 1788595121.295931 mapread 0x01caf2 thru 0x02854e (0xba5d bytes) 000768 1788595121.311127 read 0x017b81 thru 0x019afd (0x1f7d bytes) 000769 1788595121.317415 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x28a1b): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28a1b): Mirror 4: 0x550ca377 (stale) CRC-32 checksum value for chunk [0, 0x28a1b): Mirror 1: 0xf4f045e0 Mirror 5: 0xf4f045e0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28a1b: skipped 000770 1788595122.278033 mapread 0x020b92 thru 0x021b71 (0xfe0 bytes) 000773 1788595122.289251 mapwrite 0x0355dc thru 0x039348 (0x3d6d bytes) 000774 1788595122.527703 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39349): Mirror 5: 0x8522e288 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39349): Mirror 5: 0x8522e288 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39349: skipped 000775 1788595122.889519 mapread 0x0302b1 thru 0x039348 (0x9098 bytes) 000776 1788595122.907921 read 0x02f406 thru 0x0321c4 (0x2dbf bytes) 000777 1788595122.908751 mapread 0x0188e8 thru 0x026548 (0xdc61 bytes) 000778 1788595122.934428 mapread 0x01a777 thru 0x01e39e (0x3c28 bytes) 000779 1788595122.935426 read 0x00c76f thru 0x011370 (0x4c02 bytes) 000781 1788595122.956654 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ad:0x0]: Invalid argument (22) 000782 1788595123.134919 trunc from 0x039349 to 0x03509d (0x42ab bytes) 000783 1788595123.152446 write 0x025d5f thru 0x03309d (0xd33f bytes) 000784 1788595123.161324 mapwrite 0x0335a2 thru 0x035d4d (0x27ac bytes) 000785 1788595123.287700 mapread 0x006e52 thru 0x0111ff (0xa3ae bytes) 000787 1788595123.310962 trunc from 0x035d4e to 0x0060e5 (0x2fc68 bytes) 000789 1788595123.336418 trunc from 0x0060e6 to 0x0128ce (0xc7e9 bytes) 000792 1788595123.365103 mapread 0x006b88 thru 0x0128ce (0xbd47 bytes) 000793 1788595123.382409 mapwrite 0x002542 thru 0x010eb9 (0xe978 bytes) 000794 1788595123.458125 trunc from 0x0128cf to 0x01b4d3 (0x8c05 bytes) 000795 1788595123.472489 mapread 0x008a62 thru 0x00bdcb (0x336a bytes) 000797 1788595123.473801 write 0x010be9 thru 0x0193c5 (0x87dd bytes) 000799 1788595123.484139 read 0x01022f thru 0x013a62 (0x3834 bytes) 000800 1788595123.485199 trunc from 0x01b4d4 to 0x03a750 (0x1f27d bytes) 000802 1788595123.512968 read 0x01a41a thru 0x01bad6 (0x16bd bytes) 000805 1788595123.538597 trunc from 0x03a751 to 0x004d3f (0x35a11 bytes) 000806 1788595123.580466 write 0x005322 thru 0x00cc3a (0x7919 bytes) 000808 1788595123.585348 punch from 0x008127 to 0x00cc3a (0x4b14 bytes) 000809 1788595123.602532 trunc from 0x00cc3b to 0x0345ac (0x27972 bytes) 000811 1788595123.624070 punch from 0x0249a2 to 0x02a54f (0x5bae bytes) 000812 1788595123.627741 mapwrite 0x01cbed thru 0x02131f (0x4733 bytes) 000814 1788595123.698620 punch from 0x02c541 to 0x032d56 (0x6816 bytes) 000815 1788595123.702777 trunc from 0x0345ad to 0x03ce99 (0x88ed bytes) 000816 1788595123.723088 write 0x01791c thru 0x0247fb (0xcee0 bytes) 000818 1788595123.737190 trunc from 0x03ce9a to 0x018f85 (0x23f14 bytes) 000821 1788595123.754255 mapread 0x007635 thru 0x00bcb9 (0x4685 bytes) 000822 1788595123.777929 read 0x00189f thru 0x0113cd (0xfb2f bytes) 000823 1788595123.786608 mapread 0x018854 thru 0x018f85 (0x732 bytes) 000824 1788595123.790907 MIRROR_EXTEND to 2 mirrors 000826 1788595124.689557 punch from 0x012abc to 0x013cea (0x122f bytes) 000827 1788595124.779827 write 0x0171de thru 0x0242cc (0xd0ef bytes) 000829 1788595124.803512 punch from 0x007d7e to 0x008943 (0xbc6 bytes) 000830 1788595124.828787 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x242cd): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x242cd): Mirror 2: 0xbf70e568 (stale) CRC-32 checksum value for chunk [0, 0x242cd): Mirror 1: 0xe7c544d1 Mirror 3: 0xe7c544d1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x242cd: skipped 000831 1788595126.166794 mapwrite 0x001480 thru 0x00a410 (0x8f91 bytes) 000832 1788595126.362211 mapread 0x000228 thru 0x00c026 (0xbdff bytes) 000833 1788595126.370168 mapread 0x00e390 thru 0x01012e (0x1d9f bytes) 000834 1788595126.371201 mapread 0x003a1a thru 0x00bb48 (0x812f bytes) 000836 1788595126.373151 trunc from 0x0242cd to 0x00b57b (0x18d51 bytes) 000837 1788595126.398734 MIRROR_RESYNC 3 mirrors 000838 1788595127.337199 MIRROR_RESYNC 3 mirrors 000840 1788595127.846089 mapwrite 0x0087e1 thru 0x00e8d0 (0x60f0 bytes) 000841 1788595128.079453 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xe8d1): Mirror 3: 0x59614803 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe8d1): Mirror 3: 0x59614803 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe8d1: skipped 000842 1788595128.363847 trunc from 0x00e8d1 to 0x01fda2 (0x114d2 bytes) 000843 1788595128.396301 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1fda3): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fda3): Mirror 3: 0xbd7d2f49 (stale) CRC-32 checksum value for chunk [0, 0x1fda3): Mirror 1: 0x4c05fa18 Mirror 4: 0x4c05fa18 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fda3: skipped 000844 1788595129.408858 read 0x01a2bc thru 0x01fda2 (0x5ae7 bytes) 000845 1788595129.416182 trunc from 0x01fda3 to 0x0184dc (0x78c6 bytes) 000846 1788595129.520448 punch from 0x008857 to 0x015b37 (0xd2e1 bytes) 000847 1788595129.526422 mapread 0x0123cc thru 0x01297e (0x5b3 bytes) 000848 1788595129.536561 punch from 0x014466 to 0x0155ef (0x118a bytes) 000849 1788595129.549750 mapread 0x0176d9 thru 0x0184dc (0xe04 bytes) 000850 1788595129.567411 read 0x00e729 thru 0x017357 (0x8c2f bytes) 000851 1788595129.585545 mapread 0x002c8d thru 0x0120d3 (0xf447 bytes) 000852 1788595129.586780 punch from 0x0003db to 0x00bca9 (0xb8cf bytes) 000854 1788595129.600522 read 0x00957c thru 0x01623b (0xccc0 bytes) 000855 1788595129.628237 read 0x00df33 thru 0x015d23 (0x7df1 bytes) 000856 1788595129.629102 write 0x03fa7e thru 0x03ffff (0x582 bytes) 000857 1788595129.645478 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xb0eb496c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xb0eb496c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000858 1788595130.028522 mapread 0x00790b thru 0x010505 (0x8bfb bytes) 000859 1788595130.059947 trunc from 0x040000 to 0x03b30c (0x4cf3 bytes) 000860 1788595130.086418 punch from 0x01b8a2 to 0x0213e9 (0x5b48 bytes) 000862 1788595130.094467 trunc from 0x03b30d to 0x03c961 (0x1655 bytes) 000863 1788595130.138160 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ad:0x0]: Invalid argument (22) 000865 1788595130.345724 trunc from 0x03c962 to 0x01d7ca (0x1f197 bytes) 000866 1788595130.395458 mapread 0x00f8d5 thru 0x01cd3f (0xd46b bytes) 000868 1788595130.417227 mapread 0x00d4ac thru 0x017085 (0x9bda bytes) 000869 1788595130.418101 mapread 0x012a26 thru 0x01556c (0x2b47 bytes) 000871 1788595130.420649 mapwrite 0x0397ee thru 0x03ffff (0x6812 bytes) truncating to largest ever: 0x3e84b 000873 1788595130.519114 trunc from 0x040000 to 0x03e84a (0x17b5 bytes) 000875 1788595130.554690 trunc from 0x03e84b to 0x0245d4 (0x1a276 bytes) 000876 1788595130.605405 write 0x008699 thru 0x010101 (0x7a69 bytes) 000877 1788595130.617574 trunc from 0x0245d5 to 0x02cd37 (0x8763 bytes) 000878 1788595130.637499 punch from 0x00cd5d to 0x01abbd (0xde61 bytes) 000880 1788595130.652731 trunc from 0x02cd38 to 0x034dd1 (0x809a bytes) 000882 1788595130.668008 mapwrite 0x00e807 thru 0x015247 (0x6a41 bytes) 000886 1788595130.717609 trunc from 0x034dd2 to 0x02b7e8 (0x95e9 bytes) 000887 1788595130.736379 mapread 0x016a98 thru 0x023bcf (0xd138 bytes) 000888 1788595130.757355 trunc from 0x02b7e9 to 0x029728 (0x20c0 bytes) 000889 1788595130.776153 mapread 0x026f80 thru 0x029728 (0x27a9 bytes) 000891 1788595130.785516 punch from 0x0148e2 to 0x02086f (0xbf8e bytes) 000892 1788595130.789768 trunc from 0x029729 to 0x0240e8 (0x5640 bytes) 000893 1788595130.821966 punch from 0x023c36 to 0x0240e8 (0x4b3 bytes) 000895 1788595130.828951 write 0x0317ac thru 0x03626a (0x4abf bytes) 000897 1788595130.831815 write 0x0316e4 thru 0x03a636 (0x8f53 bytes) 000898 1788595130.837961 MIRROR_EXTEND to 2 mirrors 000899 1788595131.833653 read 0x01af61 thru 0x01fcf6 (0x4d96 bytes) 000900 1788595131.850120 punch from 0x02f727 to 0x0332ab (0x3b85 bytes) 000902 1788595131.940151 read 0x032fd9 thru 0x03952b (0x6553 bytes) 000903 1788595131.964905 write 0x0285b8 thru 0x02ebb7 (0x6600 bytes) 000904 1788595131.965961 mapread 0x03567e thru 0x03a636 (0x4fb9 bytes) 000905 1788595131.966825 read 0x02449b thru 0x030849 (0xc3af bytes) 000906 1788595131.967350 mapread 0x0308e7 thru 0x03a636 (0x9d50 bytes) 000907 1788595131.977043 write 0x0199d5 thru 0x022713 (0x8d3f bytes) 000908 1788595131.979593 mapwrite 0x00f636 thru 0x0104cc (0xe97 bytes) 000910 1788595132.069655 trunc from 0x03a637 to 0x02bc3b (0xe9fb bytes) 000911 1788595132.096818 read 0x021af1 thru 0x021e53 (0x363 bytes) 000912 1788595132.098043 punch from 0x00c63c to 0x017b44 (0xb509 bytes) 000914 1788595132.107387 trunc from 0x02bc3c to 0x023fe5 (0x7c56 bytes) 000916 1788595132.127094 trunc from 0x023fe6 to 0x01b6df (0x8906 bytes) 000917 1788595132.144650 write 0x01ca88 thru 0x029883 (0xcdfc bytes) 000918 1788595132.153417 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ad:0x0]: Invalid argument (22) 000921 1788595132.312534 write 0x031eb5 thru 0x03ffff (0xe14b bytes) 000922 1788595132.317157 mapread 0x0027a1 thru 0x00b1bb (0x8a1b bytes) 000923 1788595132.351661 mapwrite 0x008e36 thru 0x012bb6 (0x9d81 bytes) 000925 1788595132.415123 mapwrite 0x036feb thru 0x03ffff (0x9015 bytes) 000926 1788595132.470037 write 0x01290d thru 0x01b6a4 (0x8d98 bytes) 000927 1788595132.474713 write 0x02c19c thru 0x033a96 (0x78fb bytes) 000930 1788595132.481637 mapwrite 0x03d968 thru 0x03ffff (0x2698 bytes) 000931 1788595132.523627 punch from 0x037a27 to 0x03ffff (0x85d9 bytes) 000936 1788595132.532291 trunc from 0x040000 to 0x0033aa (0x3cc55 bytes) 000937 1788595132.546045 trunc from 0x0033ab to 0x017043 (0x13c99 bytes) 000940 1788595132.563417 mapwrite 0x0097d6 thru 0x019289 (0xfab4 bytes) 000942 1788595132.623124 write 0x02a122 thru 0x039d58 (0xfc37 bytes) 000943 1788595132.625417 mapwrite 0x030a3d thru 0x03e8a3 (0xde67 bytes) 000944 1788595132.724941 mapread 0x016222 thru 0x0228c4 (0xc6a3 bytes) 000946 1788595132.736288 write 0x03d3c0 thru 0x03ffff (0x2c40 bytes) 000947 1788595132.739973 write 0x01bacd thru 0x022932 (0x6e66 bytes) 000953 1788595132.747410 read 0x017927 thru 0x025f92 (0xe66c bytes) 000955 1788595132.755146 mapread 0x00a7f7 thru 0x019b1e (0xf328 bytes) 000957 1788595132.758127 read 0x01f5fa thru 0x02194a (0x2351 bytes) 000958 1788595132.758966 mapread 0x032be3 thru 0x039e9c (0x72ba bytes) 000962 1788595132.765356 read 0x01ca08 thru 0x01d488 (0xa81 bytes) 000963 1788595132.766103 write 0x009ec8 thru 0x0134e5 (0x961e bytes) 000964 1788595132.767818 read 0x0021fe thru 0x00249d (0x2a0 bytes) 000965 1788595132.768486 read 0x014b86 thru 0x019fdb (0x5456 bytes) 000967 1788595132.770636 mapwrite 0x008914 thru 0x013b42 (0xb22f bytes) 000968 1788595132.832436 MIRROR_EXTEND to 2 mirrors 000969 1788595133.572730 punch from 0x00b96f to 0x0160a2 (0xa734 bytes) 000970 1788595133.619966 write 0x00fd05 thru 0x011a1c (0x1d18 bytes) 000971 1788595133.629356 write 0x017f6c thru 0x023b4e (0xbbe3 bytes) 000972 1788595133.636902 trunc from 0x040000 to 0x034860 (0xb79f bytes) 000973 1788595133.660915 trunc from 0x034861 to 0x036814 (0x1fb4 bytes) 000974 1788595133.673966 mapwrite 0x006b20 thru 0x008d4e (0x222f bytes) 000975 1788595133.742274 mapread 0x00bf64 thru 0x00d26d (0x130a bytes) 000978 1788595133.743986 trunc from 0x036815 to 0x02d528 (0x92ec bytes) 000980 1788595133.762119 trunc from 0x02d529 to 0x01c7ff (0x10d29 bytes) 000981 1788595133.778028 mapwrite 0x01a04c thru 0x01a0e4 (0x99 bytes) 000982 1788595133.808810 read 0x00ef63 thru 0x012905 (0x39a3 bytes) 000984 1788595133.814178 mapwrite 0x02f465 thru 0x03bcc5 (0xc861 bytes) 000985 1788595133.884974 trunc from 0x03bcc6 to 0x017e26 (0x23e9f bytes) 000986 1788595133.906406 mapread 0x00349e thru 0x009266 (0x5dc9 bytes) 000987 1788595133.907344 read 0x014855 thru 0x017e26 (0x35d2 bytes) 000988 1788595133.922168 read 0x000277 thru 0x004bae (0x4938 bytes) 000989 1788595133.922767 punch from 0x0013cd to 0x00976d (0x83a1 bytes) 000993 1788595133.940301 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ad:0x0]: Invalid argument (22) 000994 1788595134.067050 trunc from 0x017e27 to 0x033bc8 (0x1bda2 bytes) 000996 1788595134.091171 punch from 0x01f86f to 0x020f28 (0x16ba bytes) 000998 1788595134.098922 trunc from 0x033bc9 to 0x02adfa (0x8dce bytes) 000999 1788595134.117460 trunc from 0x02adfb to 0x026773 (0x4687 bytes) 001000 1788595134.140274 trunc from 0x026774 to 0x01222f (0x14544 bytes) All operations completed A-OK! PASS 70a (96s) == sanity-flr test 71: check mirror extend parent fid ==== 03:58:59 (1788595139) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 03:59:03 (1788595143) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ee:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.263096 s, 8.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ee:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134515 s, 7.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ef:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ee:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ef:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ee:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d4:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 03:59:12 (1788595152) Starting client: oleg359-client.virtnet: -o user_xattr,flock 192.168.203.159@tcp:/lustre /mnt/lustre2 Starting client: oleg359-client.virtnet: -o user_xattr,flock 192.168.203.159@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 3580544 .. Extending file size to 6645248 .. failed Extending file size to 6741056 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6498272 .. Extending file size to 8238368 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6679616 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8726688 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8293056 .. Extending file size to 8741568 .. done Extending file size to 9155264 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8814176 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 57412 57413 57414 57417 57419 192.168.203.159@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg359-client.virtnet /mnt/lustre2 (opts:) 192.168.203.159@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg359-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 04:00:34 (1788595234) Starting client: oleg359-client.virtnet: -o user_xattr,flock 192.168.203.159@tcp:/lustre /mnt/lustre2 Starting client: oleg359-client.virtnet: -o user_xattr,flock 192.168.203.159@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3148960 .. Extending file size to 4661472 .. failed Extending file size to 5937344 .. Extending file size to 6957696 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8971168 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7278496 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7398368 .. Extending file size to 8001664 .. Extending file size to 8440704 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8582656 .. done Extending file size to 8781344 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8785952 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 58997 58999 59000 59001 59003 failed 192.168.203.159@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg359-client.virtnet /mnt/lustre2 (opts:) 192.168.203.159@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg359-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (80s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 04:01:54 (1788595314) Starting client: oleg359-client.virtnet: -o user_xattr,flock 192.168.203.159@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0908635 s, 23.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.34966 s, 6.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x23cb5de1 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.159@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg359-client.virtnet /mnt/lustre2 (opts:) PASS 200c (15s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 04:02:10 (1788595330) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0994519 s, 21.1 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 04:02:17 (1788595337) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x202:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x305:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x305:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x306:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2e9:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 04:02:24 (1788595344) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.160302 s, 13.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x307:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ea:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2eb:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 04:02:31 (1788595351) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.244655 s, 8.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x308:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x309:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ec:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x308:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x309:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ec:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ed:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 04:02:39 (1788595359) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.139076 s, 15.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ee:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ee:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ef:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 04:03:12 (1788595392) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg359-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110503 s, 9.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30b:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30b:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30c:0x0] } PASS 204d (57s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 04:04:09 (1788595449) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg359-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.385593 s, 5.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (62s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 04:05:11 (1788595511) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg359-server mds-ost sync done. Creating new pool test_204f oleg359-server: Pool lustre.test_204f created Adding targets to pool oleg359-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg359-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.154454 s, 13.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg359-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg359-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg359-server: Pool lustre.test_204f destroyed PASS 204f (75s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 04:06:27 (1788595587) PASS 205a (6s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 04:06:33 (1788595593) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 04:06:40 (1788595600) oleg359-server: Pool lustre.test_206 created Adding targets to pool oleg359-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg359-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg359-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg359-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg359-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 04:07:05 (1788595625) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926345 s, 11.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433179 s, 24.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xc7f5bb0e Mirror 3: 0xc7f5bb0e PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 04:07:16 (1788595636) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 04:07:20 (1788595640) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 04:07:23 (1788595643) Starting client: oleg359-client.virtnet: -o user_xattr,flock 192.168.203.159@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.12042 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1f50000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1550000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1550000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1550000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x13e0000 Read this (4 bytes): BBBB 192.168.203.159@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg359-client.virtnet /mnt/lustre2 (opts:) PASS 209a (34s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 04:07:57 (1788595677) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0351907 s, 233 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 04:08:04 (1788595684) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687666 s, 15.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x227 lmm_fid: [0x200000404:0x227:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x227 lmm_fid: [0x200000404:0x227:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fd:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834471 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 04:08:06.000000000 -0400 Modify: 2026-09-05 04:08:06.000000000 -0400 Change: 2026-09-05 04:08:07.000000000 -0400 Birth: 2026-09-05 04:08:05.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 04:08:12 (1788595692) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 04:08:16 (1788595696) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.454719 s, 90.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0907716 s, 45.1 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 5 04:08 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x229:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 5 04:08 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 04:08:25 (1788595705) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 04:08:32 (1788595712) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1734 sec =========== 04:08:41 (1788595721) === sanity-flr: start cleanup 04:08:42 (1788595722) === === sanity-flr: finish cleanup 04:08:49 (1788595729) ===