-----============= acceptance-small: sanity-flr ============----- Fri Sep 4 20:36:44 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg406-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 20:37:07 (1788568627) === oleg406-client.virtnet: executing check_config_client /mnt/lustre oleg406-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg406-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9b1383209000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9b1383209000.idle_timeout=debug disable quota as required oleg406-server: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 20:37:55 (1788568675) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7669] == sanity-flr test 0a: lfs mirror create with -N option == 20:37:58 (1788568678) PASS 0a (22s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 20:38:20 (1788568700) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 20:38:24 (1788568704) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (5s) == sanity-flr test 0d: lfs mirror extend with -N option == 20:38:29 (1788568709) PASS 0d (31s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 20:39:00 (1788568740) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 20:39:04 (1788568744) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 20:39:07 (1788568747) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9b1383209000.stats=clear osc.lustre-OST0001-osc-ffff9b1383209000.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 20:39:17 (1788568757) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (14s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 20:39:32 (1788568772) PASS 0j (11s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 20:39:42 (1788568782) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 20:39:46 (1788568786) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 20:39:57 (1788568797) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 20:40:06 (1788568806) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 20:40:14 (1788568814) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 20:40:17 (1788568817) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 20:40:24 (1788568824) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 20:40:34 (1788568834) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 20:40:41 (1788568841) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294708 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100736 s, 10.4 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 20:40:51 (1788568851) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 0.720248 s, 27.7 MB/s PASS 22 (11s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 20:41:02 (1788568862) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (13s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 20:41:15 (1788568875) Stopping /mnt/lustre-ost1 (opts:) on oleg406-server oleg406-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b1383209000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b1383209000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg406-server: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg406-server: oleg406-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg406-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg406-server oleg406-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b1383209000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b1383209000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg406-server: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg406-server: oleg406-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg406-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (86s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 20:42:41 (1788568961) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.326256 s, 16.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg406-server oleg406-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b1383209000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b1383209000.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg406-server: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg406-server: oleg406-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg406-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (53s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 20:43:34 (1788569014) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg406-server oleg406-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b1383209000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b1383209000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg406-server: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg406-server: oleg406-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg406-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg406-server oleg406-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b1383209000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b1383209000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg406-server: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg406-server: oleg406-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg406-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (89s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 20:45:03 (1788569103) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 6.52375 s, 8.0 MB/s -rw-r--r-- 1 root root 52428800 Sep 4 20:45 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.366s user 0m0.000s sys 0m0.180s Stopping /mnt/lustre-ost1 (opts:) on oleg406-server oleg406-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b1383209000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b1383209000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.434s user 0m0.001s sys 0m0.213s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg406-server: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg406-server: oleg406-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg406-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg406-server oleg406-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b1383209000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b1383209000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.455s user 0m0.000s sys 0m0.192s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg406-server: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg406-server: oleg406-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg406-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg406-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b1383209000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b1383209000.ost_server_uuid in FULL state after 0 sec PASS 33b (112s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 20:46:55 (1788569215) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 20:46:59 (1788569219) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 20:47:04 (1788569224) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 20:47:08 (1788569228) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 20:47:16 (1788569236) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.714196 s, 17.6 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 20:47:30 (1788569250) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039247 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313932 s, 33.4 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 20:47:37 (1788569257) PASS 36c (10s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 20:47:47 (1788569267) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000990221 s, 1.0 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00575465 s, 178 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00109496 s, 935 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00242207 s, 423 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 20:47:59 (1788569279) create test files with size 15758420 .. 5610dfb795dab6c2a32545ba33793029 - fd2c9cdb72d139c154a90fcce7df6b83 - 6dd897dda00c7e8552ce73f642b60313 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg406-client.virtnet /mnt/lustre (opts:) Starting client: oleg406-client.virtnet: -o user_xattr,flock 192.168.204.106@tcp:/lustre /mnt/lustre PASS 37 (30s) == sanity-flr test 38: resync ============================ 20:48:30 (1788569310) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (17s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 20:48:46 (1788569326) PASS 39 (9s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 20:48:55 (1788569335) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204765 s, 5.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xba:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd4:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260233 s, 40.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd5:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd6:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (12s) == sanity-flr test 41: lfs mirror resync check =========== 20:49:07 (1788569347) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.146502 s, 14.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.600145 s, 7.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.205301 s, 10.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (20s) == sanity-flr test 42: lfs mirror verify ================= 20:49:28 (1788569368) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 20:49:32 (1788569372) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 20:49:36 (1788569376) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362658 s, 28.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 20:49:45 (1788569385) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 20:49:48 (1788569388) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdd:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc3:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc3:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 20:49:59 (1788569399) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.294694 s, 35.6 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg406-client.virtnet /mnt/lustre (opts:) Starting client: oleg406-client.virtnet: -o user_xattr,flock 192.168.204.106@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 20:50:12 (1788569412) llite.lustre-ffff9b1385eb1000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.271216 s, 38.7 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg406-client.virtnet /mnt/lustre (opts:) Starting client: oleg406-client.virtnet: -o user_xattr,flock 192.168.204.106@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9b1385462800.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 20:50:28 (1788569428) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (54s) == sanity-flr test 46: Verify setstripe --copy option ==== 20:51:21 (1788569481) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (44s) == sanity-flr test 47: Verify mirror obj alloc =========== 20:52:05 (1788569525) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 20:52:09 (1788569529) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.238306 s, 13.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.136614 s, 23.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x2754890d (nosync,stale) Mirror 2: 0x2754890d (nosync,stale) ** verify mirror 2 doesn't change original checksum: 84a0a95eed78e881d3895fce8f07b5e3 - mirror 1 checksum: 040da65b38c0e18ec4ca29246108a950 - mirror 2 checksum: 84a0a95eed78e881d3895fce8f07b5e3 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-04T20:52:13-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-04T20:52:13-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd9:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 84a0a95eed78e881d3895fce8f07b5e3 - mirror 1 checksum: 040da65b38c0e18ec4ca29246108a950 - mirror 2 checksum: 040da65b38c0e18ec4ca29246108a950 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-04T20:52:13-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-04T20:52:13-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd9:0x0] } PASS 48 (12s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 20:52:22 (1788569542) Failing mds1 on oleg406-server Stopping /mnt/lustre-mds1 (opts:) on oleg406-server 20:52:29 (1788569549) shut down facet: mds1 facet_host: oleg406-server facet_failover_host: oleg406-server Failover mds1 to oleg406-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg406-server: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-MDT0000 20:53:04 (1788569584) targets are mounted 20:53:04 (1788569584) facet_failover done oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412324 s, 25.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } PASS 50A (62s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 20:53:24 (1788569604) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 20:53:29 (1788569609) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (5s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 20:53:33 (1788569613) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0123446 s, 1.3 MB/s PASS 50c (10s) == sanity-flr test 50d: mirror rsync keep holes ========== 20:53:44 (1788569624) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 20:53:47 (1788569627) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0104882 s, 391 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0753167 s, 54.4 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 20:53:56 (1788569636) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (27s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 20:54:22 (1788569662) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (34s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 20:54:56 (1788569696) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 20:55:21 (1788569721) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 20:55:46 (1788569746) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 20:55:53 (1788569753) mirror create pid 54387 mirror split pid 54389 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54387 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54389 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 20:57:01 (1788569821) Seed set to 9823 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000004 1788569824.093321 write 0x01a54f thru 0x0298c5 (0xf377 bytes) 000006 1788569824.096963 read 0x017c2a thru 0x020f94 (0x936b bytes) truncating to largest ever: 0xc76d 000007 1788569824.102430 trunc from 0x0298c6 to 0x00c76c (0x1d159 bytes) truncating to largest ever: 0x34492 000009 1788569824.118958 trunc from 0x00c76d to 0x034491 (0x27d25 bytes) 000010 1788569824.143358 mapwrite 0x02cb0b thru 0x03b8ae (0xeda4 bytes) 000012 1788569824.370523 mapwrite 0x011864 thru 0x01a905 (0x90a2 bytes) 000017 1788569824.565117 mapwrite 0x007518 thru 0x007917 (0x400 bytes) 000018 1788569824.652391 MIRROR_EXTEND to 2 mirrors 000021 1788569826.126794 mapread 0x022eda thru 0x02b7af (0x88d6 bytes) 000022 1788569826.179530 punch from 0x011aef to 0x018e71 (0x7383 bytes) 000023 1788569826.359847 read 0x039f28 thru 0x03b8ae (0x1987 bytes) 000024 1788569826.367334 mapread 0x02ea36 thru 0x03501e (0x65e9 bytes) 000026 1788569826.384791 read 0x01c8f2 thru 0x0244bd (0x7bcc bytes) 000027 1788569826.396860 read 0x02afec thru 0x030d11 (0x5d26 bytes) 000028 1788569826.397443 punch from 0x012285 to 0x016deb (0x4b67 bytes) 000031 1788569826.404909 mapread 0x01d55a thru 0x029832 (0xc2d9 bytes) 000032 1788569826.413904 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b8af): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b8af): Mirror 2: 0x9860e5d9 (stale) CRC-32 checksum value for chunk [0, 0x3b8af): Mirror 1: 0xfee48a72 Mirror 3: 0xfee48a72 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b8af: skipped 000033 1788569827.877162 write 0x0221ae thru 0x0300e0 (0xdf33 bytes) 000034 1788569827.977146 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3b8af): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b8af): Mirror 2: 0x9860e5d9 (stale) Mirror 3: 0xfee48a72 (stale) CRC-32 checksum value for chunk [0, 0x3b8af): Mirror 1: 0x65911158 Mirror 4: 0x65911158 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b8af: skipped truncating to largest ever: 0x3efd3 000035 1788569829.635241 trunc from 0x03b8af to 0x03efd2 (0x3724 bytes) 000036 1788569829.744425 punch from 0x017ff5 to 0x01af22 (0x2f2e bytes) 000037 1788569829.750278 punch from 0x02ccd6 to 0x03a168 (0xd493 bytes) 000038 1788569829.763980 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3efd3): Mirror 3: 0xd6017709 (stale) Mirror 4: 0x8ce2c3b6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3efd3): Mirror 3: 0xd6017709 (stale) Mirror 4: 0x8ce2c3b6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3efd3: skipped 000039 1788569830.248463 punch from 0x023b96 to 0x02f9ba (0xbe25 bytes) 000040 1788569830.260999 write 0x01ad3d thru 0x02a8e4 (0xfba8 bytes) 000041 1788569830.296844 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3efd3): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3efd3): Mirror 3: 0xd6017709 (stale) Mirror 4: 0x8ce2c3b6 (stale) CRC-32 checksum value for chunk [0, 0x3efd3): Mirror 1: 0x9b6bb3ba Mirror 5: 0x9b6bb3ba lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3efd3: skipped 000043 1788569832.413990 mapread 0x0124da thru 0x01fdb1 (0xd8d8 bytes) 000044 1788569832.465410 trunc from 0x03efd3 to 0x013e92 (0x2b140 bytes) 000045 1788569832.591658 punch from 0x004565 to 0x009afb (0x5597 bytes) 000046 1788569832.604336 mapread 0x011998 thru 0x013e92 (0x24fb bytes) 000047 1788569832.616877 trunc from 0x013e93 to 0x031625 (0x1d793 bytes) 000048 1788569832.668300 trunc from 0x031626 to 0x02d6cf (0x3f56 bytes) 000049 1788569832.714171 trunc from 0x02d6d0 to 0x0341a2 (0x6ad3 bytes) 000050 1788569832.759682 write 0x009058 thru 0x00e200 (0x51a9 bytes) 000057 1788569832.776345 read 0x0067ad thru 0x006cbd (0x511 bytes) 000059 1788569832.783700 mapread 0x020ed7 thru 0x028ac0 (0x7bea bytes) 000060 1788569832.815817 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x341a3): Mirror 4: 0x3bbabe7c (stale) Mirror 5: 0x8624a19b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x341a3): Mirror 4: 0x3bbabe7c (stale) Mirror 5: 0x8624a19b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x341a3: skipped 000061 1788569833.181270 MIRROR_RESYNC 3 mirrors 000062 1788569834.586175 mapread 0x02ec55 thru 0x031863 (0x2c0f bytes) 000064 1788569834.610686 trunc from 0x0341a3 to 0x021b61 (0x12641 bytes) 000065 1788569834.731879 write 0x012f82 thru 0x01fc13 (0xcc92 bytes) 000066 1788569834.753372 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x21b62): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21b62): Mirror 4: 0xaef7ef1f (stale) Mirror 5: 0xaef7ef1f (stale) CRC-32 checksum value for chunk [0, 0x21b62): Mirror 1: 0x3d3f5a20 Mirror 6: 0x3d3f5a20 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21b62: skipped 000067 1788569836.836425 write 0x038bb0 thru 0x0397c5 (0xc16 bytes) 000068 1788569836.951740 mapread 0x000e4b thru 0x0010d7 (0x28d bytes) 000070 1788569836.968059 mapread 0x03006d thru 0x0397c5 (0x9759 bytes) 000071 1788569837.000756 write 0x01ba8d thru 0x025c40 (0xa1b4 bytes) 000072 1788569837.015546 mapwrite 0x0007f6 thru 0x00a4ed (0x9cf8 bytes) 000074 1788569837.131433 trunc from 0x0397c6 to 0x010dd8 (0x289ed bytes) 000076 1788569837.162110 write 0x03c6bb thru 0x03ffff (0x3945 bytes) 000077 1788569837.164169 punch from 0x02e76b to 0x03bdd6 (0xd66c bytes) 000078 1788569837.179861 punch from 0x03800a to 0x03c91a (0x4911 bytes) 000079 1788569837.193443 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x6e0511a1 (stale) Mirror 6: 0x96c42173 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x6e0511a1 (stale) Mirror 6: 0x96c42173 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000080 1788569837.897415 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x6e0511a1 (stale) Mirror 6: 0x96c42173 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xcfaab71b Mirror 7: 0xcfaab71b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000081 1788569839.776396 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x96c42173 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x96c42173 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xcfaab71b Mirror 7: 0xcfaab71b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000083 1788569840.484021 punch from 0x035865 to 0x03aff7 (0x5793 bytes) 000084 1788569840.615012 read 0x020a76 thru 0x021321 (0x8ac bytes) 000086 1788569840.628173 trunc from 0x040000 to 0x034381 (0xbc7e bytes) 000087 1788569840.652314 read 0x0090f9 thru 0x016ac6 (0xd9ce bytes) 000088 1788569840.664914 trunc from 0x034382 to 0x02100e (0x13373 bytes) 000090 1788569840.692086 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2100f): Mirror 6: 0x1f06692d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2100f): Mirror 6: 0x1f06692d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2100f: skipped 000091 1788569841.078636 mapwrite 0x01156d thru 0x01b059 (0x9aed bytes) 000092 1788569841.168845 MIRROR_RESYNC 2 mirrors 000098 1788569842.019045 punch from 0x016401 to 0x02100e (0xac0e bytes) 000099 1788569842.157091 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) 000100 1788569842.478321 read 0x00b193 thru 0x00c59d (0x140b bytes) 000101 1788569842.494452 punch from 0x00560d to 0x00730a (0x1cfe bytes) 000103 1788569842.507930 write 0x0207ff thru 0x02cc32 (0xc434 bytes) 000104 1788569842.520784 mapread 0x01d851 thru 0x024365 (0x6b15 bytes) 000105 1788569842.561184 punch from 0x00f0f7 to 0x019760 (0xa66a bytes) 000106 1788569842.599270 write 0x015021 thru 0x0164d1 (0x14b1 bytes) 000107 1788569842.617538 mapwrite 0x03f8a9 thru 0x03ffff (0x757 bytes) 000108 1788569842.715010 mapread 0x0173ad thru 0x02649d (0xf0f1 bytes) 000109 1788569842.724947 trunc from 0x040000 to 0x029d26 (0x162d9 bytes) 000111 1788569842.742364 read 0x016300 thru 0x01e41c (0x811d bytes) 000112 1788569842.742958 mapread 0x002987 thru 0x008c1a (0x6294 bytes) 000113 1788569842.752427 write 0x024a60 thru 0x03244c (0xd9ed bytes) 000114 1788569842.754311 mapread 0x01e62f thru 0x0204f3 (0x1ec5 bytes) 000115 1788569842.756032 mapread 0x015dcd thru 0x01945a (0x368e bytes) 000116 1788569842.756941 mapread 0x005ce0 thru 0x00785b (0x1b7c bytes) 000119 1788569842.763717 punch from 0x029b69 to 0x0300dc (0x6574 bytes) 000120 1788569842.778268 read 0x0027d0 thru 0x009ce3 (0x7514 bytes) 000122 1788569842.780542 mapread 0x01d550 thru 0x022ba8 (0x5659 bytes) 000123 1788569842.782054 mapread 0x0207fe thru 0x024dd5 (0x45d8 bytes) 000125 1788569842.784321 mapwrite 0x001d2a thru 0x00d6d2 (0xb9a9 bytes) 000127 1788569842.852494 punch from 0x005018 to 0x00fc24 (0xac0d bytes) 000128 1788569842.859942 mapread 0x026919 thru 0x02f6a2 (0x8d8a bytes) 000129 1788569842.877840 mapwrite 0x005ac9 thru 0x012ad0 (0xd008 bytes) 000130 1788569842.977159 MIRROR_EXTEND to 2 mirrors 000132 1788569844.396945 read 0x020f7c thru 0x023cc4 (0x2d49 bytes) 000133 1788569844.418042 read 0x011ef2 thru 0x014cde (0x2ded bytes) 000134 1788569844.446527 mapread 0x011f13 thru 0x0169eb (0x4ad9 bytes) 000135 1788569844.450480 mapwrite 0x00ea7d thru 0x010c8a (0x220e bytes) 000136 1788569844.729328 mapread 0x002004 thru 0x00a5e5 (0x85e2 bytes) 000138 1788569844.867144 read 0x02922e thru 0x02b496 (0x2269 bytes) 000139 1788569844.909253 read 0x00c481 thru 0x00e30e (0x1e8e bytes) 000141 1788569844.911579 mapwrite 0x0281e2 thru 0x028d01 (0xb20 bytes) 000142 1788569844.983552 mapwrite 0x0277a9 thru 0x02de0a (0x6662 bytes) 000143 1788569845.123371 read 0x00e919 thru 0x011762 (0x2e4a bytes) 000144 1788569845.137616 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) 000146 1788569845.355584 read 0x003366 thru 0x009176 (0x5e11 bytes) 000147 1788569845.392472 mapread 0x009899 thru 0x017824 (0xdf8c bytes) 000148 1788569845.440801 write 0x02a44e thru 0x02c56b (0x211e bytes) 000150 1788569845.467488 write 0x00ffaf thru 0x012316 (0x2368 bytes) 000151 1788569845.487236 mapread 0x018b67 thru 0x01cab2 (0x3f4c bytes) 000153 1788569845.490094 mapwrite 0x00aa50 thru 0x00b93b (0xeec bytes) 000154 1788569845.596516 mapwrite 0x0396dd thru 0x03ffff (0x6923 bytes) 000155 1788569845.776110 trunc from 0x040000 to 0x00b962 (0x3469d bytes) 000156 1788569845.807624 write 0x006a32 thru 0x006d20 (0x2ef bytes) 000158 1788569845.809455 read 0x008bad thru 0x00aa30 (0x1e84 bytes) 000159 1788569845.810687 write 0x0176dd thru 0x0258a0 (0xe1c4 bytes) 000161 1788569845.814372 read 0x00715d thru 0x00b995 (0x4839 bytes) 000163 1788569845.816135 write 0x018983 thru 0x01a518 (0x1b96 bytes) 000164 1788569845.817391 read 0x01f2a3 thru 0x022ccf (0x3a2d bytes) 000165 1788569845.818318 trunc from 0x0258a1 to 0x008412 (0x1d48e bytes) 000167 1788569845.841724 trunc from 0x008413 to 0x03caad (0x3469b bytes) 000168 1788569845.865347 write 0x020a1d thru 0x02361f (0x2c03 bytes) 000169 1788569845.888842 write 0x028f18 thru 0x02d8f8 (0x49e1 bytes) 000170 1788569845.890049 mapread 0x03c934 thru 0x03caad (0x17a bytes) 000172 1788569845.893498 read 0x0229f2 thru 0x02cce1 (0xa2f0 bytes) 000174 1788569845.896200 trunc from 0x03caae to 0x000979 (0x3c134 bytes) 000175 1788569845.921902 write 0x03b470 thru 0x03ffff (0x4b90 bytes) 000176 1788569845.924852 punch from 0x01891f to 0x027c31 (0xf313 bytes) 000177 1788569845.946913 mapread 0x006d37 thru 0x00fed4 (0x919e bytes) 000178 1788569845.978772 trunc from 0x040000 to 0x0340fa (0xbf05 bytes) 000180 1788569846.017762 MIRROR_EXTEND to 2 mirrors 000181 1788569847.394188 trunc from 0x0340fb to 0x00c09b (0x2805f bytes) 000182 1788569847.559147 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xc09c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc09c): Mirror 2: 0x636f089 (stale) CRC-32 checksum value for chunk [0, 0xc09c): Mirror 1: 0x636f089 Mirror 3: 0x636f089 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc09c: skipped 000183 1788569848.633324 MIRROR_RESYNC 3 mirrors 000184 1788569849.748078 trunc from 0x00c09c to 0x00ae45 (0x1256 bytes) 000185 1788569849.829910 trunc from 0x00ae46 to 0x009b19 (0x132c bytes) 000186 1788569849.850285 trunc from 0x009b1a to 0x02dee9 (0x243d0 bytes) 000187 1788569849.877834 write 0x01fcf9 thru 0x020c80 (0xf88 bytes) 000189 1788569849.889182 write 0x021738 thru 0x02b7a0 (0xa069 bytes) 000190 1788569849.899102 trunc from 0x02deea to 0x03c642 (0xe759 bytes) 000191 1788569849.932058 mapwrite 0x01088d thru 0x01f8c0 (0xf034 bytes) 000193 1788569850.045379 write 0x02eb84 thru 0x03aa2c (0xbea9 bytes) 000194 1788569850.064280 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c643): Mirror 3: 0xf23967da (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c643): Mirror 3: 0xf23967da (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c643: skipped 000195 1788569850.481488 MIRROR_RESYNC 2 mirrors 000198 1788569851.766737 trunc from 0x03c643 to 0x00d976 (0x2eccc bytes) 000199 1788569851.843964 trunc from 0x00d977 to 0x034ae9 (0x27173 bytes) 000201 1788569851.868895 trunc from 0x034aea to 0x0327e5 (0x2304 bytes) 000202 1788569851.890938 punch from 0x00124a to 0x00781a (0x65d1 bytes) 000203 1788569851.895844 punch from 0x00ae3d to 0x012809 (0x79cd bytes) 000204 1788569851.900526 mapwrite 0x01df58 thru 0x01fb3a (0x1be3 bytes) 000206 1788569851.962031 punch from 0x02cfef to 0x0307d9 (0x37eb bytes) 000207 1788569851.967537 punch from 0x003408 to 0x013043 (0xfc3c bytes) 000208 1788569851.972334 trunc from 0x0327e6 to 0x03af80 (0x879b bytes) 000210 1788569851.991917 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3af81): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3af81): Mirror 3: 0x6f086847 (stale) CRC-32 checksum value for chunk [0, 0x3af81): Mirror 1: 0xc163387a Mirror 4: 0xc163387a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3af81: skipped 000211 1788569853.530801 mapread 0x00458c thru 0x005e6b (0x18e0 bytes) 000213 1788569853.542329 mapwrite 0x00e8cd thru 0x0149fd (0x6131 bytes) 000214 1788569853.704812 punch from 0x01f985 to 0x02e23f (0xe8bb bytes) 000216 1788569853.711792 write 0x01f239 thru 0x021387 (0x214f bytes) 000217 1788569853.722692 mapwrite 0x01959c thru 0x0250e2 (0xbb47 bytes) 000218 1788569853.802679 trunc from 0x03af81 to 0x02b191 (0xfdef bytes) 000219 1788569853.836247 trunc from 0x02b192 to 0x0179d5 (0x137bc bytes) 000221 1788569853.869596 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x179d6): Mirror 4: 0x982cec2f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x179d6): Mirror 4: 0x982cec2f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x179d6: skipped 000224 1788569854.176500 mapread 0x00e8b7 thru 0x014b66 (0x62b0 bytes) 000225 1788569854.197623 trunc from 0x0179d6 to 0x0155c6 (0x240f bytes) 000229 1788569854.219828 mapwrite 0x0035c0 thru 0x006c3f (0x3680 bytes) 000230 1788569854.285835 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x155c7): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x155c7): Mirror 4: 0x356bf0b7 (stale) CRC-32 checksum value for chunk [0, 0x155c7): Mirror 1: 0x27cd2337 Mirror 5: 0x27cd2337 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x155c7: skipped 000232 1788569855.470278 mapread 0x00a0f0 thru 0x0155c6 (0xb4d7 bytes) 000233 1788569855.546401 mapwrite 0x026a48 thru 0x031483 (0xaa3c bytes) 000234 1788569855.802927 mapread 0x00ea5b thru 0x018dd8 (0xa37e bytes) 000235 1788569855.838159 mapread 0x031136 thru 0x031483 (0x34e bytes) 000236 1788569855.839955 write 0x013ec5 thru 0x023dd3 (0xff0f bytes) 000237 1788569855.846753 trunc from 0x031484 to 0x02bd29 (0x575a bytes) 000239 1788569855.907269 mapwrite 0x01909c thru 0x01c707 (0x366c bytes) 000240 1788569855.981696 punch from 0x01bd77 to 0x02a741 (0xe9cb bytes) 000241 1788569855.988765 mapread 0x01ddb2 thru 0x02bd29 (0xdf78 bytes) 000242 1788569856.005908 mapwrite 0x034a27 thru 0x03ffff (0xb5d9 bytes) 000243 1788569856.098016 mapread 0x00e2af thru 0x01bf65 (0xdcb7 bytes) 000244 1788569856.105734 mapwrite 0x0018c5 thru 0x0073d5 (0x5b11 bytes) 000246 1788569856.201308 trunc from 0x040000 to 0x024087 (0x1bf78 bytes) 000249 1788569856.234293 trunc from 0x024088 to 0x03616c (0x120e5 bytes) 000250 1788569856.268928 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3616d): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3616d): Mirror 4: 0xd3aacaa8 (stale) Mirror 5: 0xeeb908a0 (stale) CRC-32 checksum value for chunk [0, 0x3616d): Mirror 1: 0x949bb32c Mirror 6: 0x949bb32c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3616d: skipped 000251 1788569857.913020 mapread 0x0268f3 thru 0x0340b4 (0xd7c2 bytes) 000253 1788569857.979128 punch from 0x02830a to 0x029503 (0x11fa bytes) 000254 1788569858.072026 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3616d): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3616d): Mirror 4: 0xd3aacaa8 (stale) Mirror 5: 0xeeb908a0 (stale) Mirror 6: 0x949bb32c (stale) CRC-32 checksum value for chunk [0, 0x3616d): Mirror 1: 0x949bb32c Mirror 7: 0x949bb32c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3616d: skipped 000255 1788569859.742667 MIRROR_RESYNC 5 mirrors 000257 1788569861.514728 read 0x028c8c thru 0x02b2b4 (0x2629 bytes) 000259 1788569861.525737 write 0x03c4b5 thru 0x03ffff (0x3b4b bytes) 000260 1788569861.618744 punch from 0x03aac8 to 0x03ffff (0x5538 bytes) 000262 1788569861.632466 punch from 0x038a17 to 0x03bcdc (0x32c6 bytes) 000264 1788569861.644655 mapwrite 0x024612 thru 0x03409c (0xfa8b bytes) 000265 1788569861.784648 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x756afe52 (stale) Mirror 5: 0x756afe52 (stale) Mirror 6: 0x756afe52 (stale) Mirror 7: 0x756afe52 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbf013145 Mirror 8: 0xbf013145 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000266 1788569863.543233 read 0x007d9c thru 0x015f40 (0xe1a5 bytes) 000267 1788569863.563595 write 0x031f6d thru 0x03ff44 (0xdfd8 bytes) 000269 1788569863.727444 read 0x00cd7b thru 0x01b2c0 (0xe546 bytes) 000270 1788569863.746752 trunc from 0x040000 to 0x01aa95 (0x2556a bytes) 000271 1788569863.793786 write 0x000abb thru 0x008344 (0x788a bytes) 000272 1788569863.795678 write 0x00d5ab thru 0x01146a (0x3ec0 bytes) 000273 1788569863.802386 MIRROR_RESYNC 6 mirrors 000275 1788569866.117970 MIRROR_EXTEND to 7 mirrors 000277 1788569867.463863 mapwrite 0x001079 thru 0x0109ed (0xf975 bytes) 000279 1788569867.873246 trunc from 0x01aa96 to 0x0365ef (0x1bb5a bytes) truncating to largest ever: 0x3fb82 000280 1788569867.919261 trunc from 0x0365f0 to 0x03fb81 (0x9592 bytes) 000284 1788569867.966419 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3fb82): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fb82): Mirror 4: 0xdc87f230 (stale) Mirror 5: 0xdc87f230 (stale) Mirror 6: 0xdc87f230 (stale) Mirror 7: 0xdc87f230 (stale) Mirror 8: 0xdc87f230 (stale) Mirror 9: 0xdc87f230 (stale) CRC-32 checksum value for chunk [0, 0x3fb82): Mirror 1: 0xd7b71223 Mirror 10: 0xd7b71223 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fb82: skipped 000285 1788569869.534226 read 0x006f0d thru 0x00c197 (0x528b bytes) 000286 1788569869.561927 MIRROR_RESYNC 8 mirrors 000287 1788569871.527048 trunc from 0x03fb82 to 0x03aed1 (0x4cb0 bytes) 000289 1788569871.660556 punch from 0x032f9e to 0x03aed1 (0x7f34 bytes) 000292 1788569871.667812 read 0x009171 thru 0x016167 (0xcff7 bytes) 000296 1788569871.680549 punch from 0x0115e8 to 0x01a1c3 (0x8bdc bytes) 000297 1788569871.687075 trunc from 0x03aed2 to 0x02127c (0x19c55 bytes) 000298 1788569871.718914 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2127d): Mirror 4: 0x3bd5285f (stale) Mirror 5: 0x3bd5285f (stale) Mirror 7: 0x3bd5285f (stale) Mirror 8: 0x3bd5285f (stale) Mirror 9: 0x3bd5285f (stale) Mirror 10: 0x3bd5285f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2127d): Mirror 4: 0x3bd5285f (stale) Mirror 5: 0x3bd5285f (stale) Mirror 7: 0x3bd5285f (stale) Mirror 8: 0x3bd5285f (stale) Mirror 9: 0x3bd5285f (stale) Mirror 10: 0x3bd5285f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2127d: skipped 000300 1788569872.456767 mapwrite 0x01e221 thru 0x02df37 (0xfd17 bytes) 000301 1788569872.568730 mapread 0x010c2b thru 0x014fa5 (0x437b bytes) 000302 1788569872.581752 mapwrite 0x01c130 thru 0x0202fa (0x41cb bytes) 000303 1788569872.639502 mapread 0x0255a7 thru 0x02df37 (0x8991 bytes) 000308 1788569872.643885 read 0x01dfcc thru 0x02193a (0x396f bytes) 000309 1788569872.649719 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2df38): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2df38): Mirror 4: 0xecaf6c3e (stale) Mirror 5: 0xecaf6c3e (stale) Mirror 7: 0xecaf6c3e (stale) Mirror 8: 0xecaf6c3e (stale) Mirror 9: 0xecaf6c3e (stale) Mirror 10: 0xecaf6c3e (stale) CRC-32 checksum value for chunk [0, 0x2df38): Mirror 1: 0x9d357ac1 Mirror 11: 0x9d357ac1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2df38: skipped 000310 1788569873.882406 mapread 0x02ccc3 thru 0x02df37 (0x1275 bytes) 000311 1788569873.917939 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2df38): Mirror 5: 0xecaf6c3e (stale) Mirror 7: 0xecaf6c3e (stale) Mirror 8: 0xecaf6c3e (stale) Mirror 9: 0xecaf6c3e (stale) Mirror 10: 0xecaf6c3e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2df38): Mirror 5: 0xecaf6c3e (stale) Mirror 7: 0xecaf6c3e (stale) Mirror 8: 0xecaf6c3e (stale) Mirror 9: 0xecaf6c3e (stale) Mirror 10: 0xecaf6c3e (stale) CRC-32 checksum value for chunk [0, 0x2df38): Mirror 1: 0x9d357ac1 Mirror 11: 0x9d357ac1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2df38: skipped 000314 1788569874.653208 trunc from 0x02df38 to 0x019f4c (0x13feb bytes) 000315 1788569874.800361 MIRROR_RESYNC 7 mirrors 000318 1788569876.489723 MIRROR_RESYNC 7 mirrors 000321 1788569876.945693 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) 000322 1788569877.299531 read 0x0120bf thru 0x012c1f (0xb61 bytes) 000323 1788569877.311292 MIRROR_RESYNC 6 mirrors 000324 1788569877.703552 punch from 0x0066d5 to 0x00e8ed (0x8219 bytes) 000326 1788569877.794765 write 0x024553 thru 0x03394e (0xf3fc bytes) 000328 1788569877.802873 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3394f): Mirror 5: 0x7bb90ff3 (stale) Mirror 7: 0x7bb90ff3 (stale) Mirror 8: 0x7bb90ff3 (stale) Mirror 11: 0x7bb90ff3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3394f): Mirror 5: 0x7bb90ff3 (stale) Mirror 7: 0x7bb90ff3 (stale) Mirror 8: 0x7bb90ff3 (stale) Mirror 11: 0x7bb90ff3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3394f: skipped 000329 1788569878.185502 mapread 0x02588a thru 0x0272ac (0x1a23 bytes) 000331 1788569878.205736 mapwrite 0x02d77a thru 0x03b928 (0xe1af bytes) 000332 1788569878.311087 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3b929): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b929): Mirror 5: 0x98f44216 (stale) Mirror 7: 0x98f44216 (stale) Mirror 8: 0x98f44216 (stale) Mirror 11: 0x98f44216 (stale) CRC-32 checksum value for chunk [0, 0x3b929): Mirror 1: 0xd189c1cc Mirror 12: 0xd189c1cc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b929: skipped 000333 1788569880.021796 punch from 0x013bc6 to 0x02322e (0xf669 bytes) 000334 1788569880.148603 mapread 0x00a167 thru 0x0174a6 (0xd340 bytes) 000335 1788569880.183131 punch from 0x010264 to 0x017499 (0x7236 bytes) 000337 1788569880.188251 mapread 0x002601 thru 0x012462 (0xfe62 bytes) 000340 1788569880.208781 write 0x01e7c2 thru 0x021265 (0x2aa4 bytes) 000342 1788569880.220242 trunc from 0x03b929 to 0x006026 (0x35902 bytes) 000343 1788569880.258733 MIRROR_RESYNC 6 mirrors 000344 1788569881.659878 punch from 0x004cbe to 0x006026 (0x1369 bytes) 000345 1788569881.778472 mapwrite 0x0242a8 thru 0x02763a (0x3393 bytes) 000347 1788569881.910412 read 0x01683a thru 0x02068c (0x9e53 bytes) 000349 1788569881.927477 mapread 0x007b17 thru 0x00d859 (0x5d43 bytes) 000352 1788569881.948206 trunc from 0x02763b to 0x029a3c (0x2402 bytes) 000353 1788569881.986272 write 0x03b0f6 thru 0x03ffff (0x4f0a bytes) 000354 1788569881.987529 punch from 0x017ef4 to 0x01e6d9 (0x67e6 bytes) 000356 1788569882.013870 mapwrite 0x01730d thru 0x01a0ac (0x2da0 bytes) 000358 1788569882.106767 punch from 0x01e433 to 0x02166c (0x323a bytes) 000360 1788569882.118509 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x5e325619 (stale) Mirror 7: 0x5e325619 (stale) Mirror 8: 0x5e325619 (stale) Mirror 11: 0x5e325619 (stale) Mirror 12: 0x5e325619 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfa84ffc3 Mirror 13: 0xfa84ffc3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000361 1788569883.714225 read 0x02859c thru 0x03853a (0xff9f bytes) 000362 1788569883.743110 mapwrite 0x0269df thru 0x0303da (0x99fc bytes) 000366 1788569884.015702 punch from 0x00f430 to 0x015dc6 (0x6997 bytes) 000367 1788569884.022298 punch from 0x03fa46 to 0x03ffff (0x5ba bytes) 000368 1788569884.027408 trunc from 0x040000 to 0x01e73e (0x218c1 bytes) 000369 1788569884.092115 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1e73f): Mirror 5: 0xa2c22c19 (stale) Mirror 8: 0xa2c22c19 (stale) Mirror 11: 0xa2c22c19 (stale) Mirror 12: 0xa2c22c19 (stale) Mirror 13: 0x1eca8bc3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e73f): Mirror 5: 0xa2c22c19 (stale) Mirror 8: 0xa2c22c19 (stale) Mirror 11: 0xa2c22c19 (stale) Mirror 12: 0xa2c22c19 (stale) Mirror 13: 0x1eca8bc3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e73f: skipped 000370 1788569884.536146 write 0x01747c thru 0x01fb95 (0x871a bytes) 000371 1788569884.546081 read 0x00a144 thru 0x00caee (0x29ab bytes) 000372 1788569884.557349 mapread 0x011069 thru 0x016a52 (0x59ea bytes) 000373 1788569884.585615 punch from 0x01f8d3 to 0x01fb95 (0x2c3 bytes) 000375 1788569884.604175 trunc from 0x01fb96 to 0x012eea (0xccab bytes) 000377 1788569884.634174 write 0x021f7d thru 0x03059d (0xe621 bytes) 000379 1788569884.639339 punch from 0x0231d6 to 0x028e52 (0x5c7d bytes) 000380 1788569884.660481 read 0x00c237 thru 0x01a511 (0xe2db bytes) 000382 1788569884.675375 write 0x02053c thru 0x02f4fd (0xefc2 bytes) 000383 1788569884.678750 mapwrite 0x00721c thru 0x016410 (0xf1f5 bytes) 000384 1788569884.762069 mapread 0x009e32 thru 0x00b521 (0x16f0 bytes) 000385 1788569884.762924 mapwrite 0x00faee thru 0x00ff28 (0x43b bytes) 000386 1788569884.834544 mapwrite 0x00d0c2 thru 0x015893 (0x87d2 bytes) 000388 1788569884.923484 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3059e): Mirror 5: 0xf7f20b80 (stale) Mirror 8: 0xf7f20b80 (stale) Mirror 11: 0xf7f20b80 (stale) Mirror 13: 0x2d02ff8f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3059e): Mirror 5: 0xf7f20b80 (stale) Mirror 8: 0xf7f20b80 (stale) Mirror 11: 0xf7f20b80 (stale) Mirror 13: 0x2d02ff8f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3059e: skipped 000389 1788569885.451378 write 0x02deec thru 0x02fcc1 (0x1dd6 bytes) 000390 1788569885.479230 MIRROR_RESYNC 5 mirrors 000391 1788569887.770664 read 0x00c4b8 thru 0x0136d3 (0x721c bytes) 000392 1788569887.798212 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) 000394 1788569888.450746 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) 000395 1788569889.246412 write 0x0032e3 thru 0x007ec8 (0x4be6 bytes) 000397 1788569889.379126 read 0x015db4 thru 0x0180af (0x22fc bytes) 000398 1788569889.390593 trunc from 0x03059e to 0x03a29f (0x9d02 bytes) 000399 1788569889.415953 punch from 0x02944c to 0x02b0ef (0x1ca4 bytes) 000400 1788569889.430161 mapwrite 0x014bbc thru 0x022c85 (0xe0ca bytes) 000401 1788569889.536466 read 0x008d77 thru 0x00b0d5 (0x235f bytes) 000402 1788569889.553063 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3a2a0): Mirror 11: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a2a0): Mirror 11: 0x8e131286 (stale) Mirror 13: 0x8e131286 (stale) CRC-32 checksum value for chunk [0, 0x3a2a0): Mirror 1: 0xdf04f420 Mirror 14: 0xdf04f420 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a2a0: skipped 000403 1788569890.963381 read 0x015a24 thru 0x019ebc (0x4499 bytes) 000406 1788569891.000519 mapwrite 0x037d95 thru 0x038d5b (0xfc7 bytes) 000407 1788569891.280103 trunc from 0x03a2a0 to 0x03b603 (0x1364 bytes) 000408 1788569891.305072 punch from 0x032f8f to 0x03b603 (0x8675 bytes) 000410 1788569891.311640 read 0x0154da thru 0x01f842 (0xa369 bytes) 000411 1788569891.334546 write 0x0371c5 thru 0x03ffff (0x8e3b bytes) 000412 1788569891.336194 punch from 0x01e3e6 to 0x027b08 (0x9723 bytes) 000414 1788569891.365239 mapread 0x0246cc thru 0x026bda (0x250f bytes) 000416 1788569891.386586 trunc from 0x040000 to 0x02948f (0x16b70 bytes) 000417 1788569891.413878 mapread 0x02055a thru 0x0241ef (0x3c96 bytes) 000418 1788569891.453836 mapwrite 0x01b484 thru 0x0291b0 (0xdd2d bytes) 000420 1788569891.599692 punch from 0x01d88e to 0x02948f (0xbc02 bytes) 000424 1788569891.623892 trunc from 0x029490 to 0x037de3 (0xe954 bytes) 000425 1788569891.658399 read 0x002485 thru 0x00edc1 (0xc93d bytes) 000426 1788569891.660028 mapread 0x031a79 thru 0x0324af (0xa37 bytes) 000428 1788569891.683226 mapwrite 0x033cc3 thru 0x03ffff (0xc33d bytes) 000429 1788569891.815104 read 0x011a07 thru 0x016681 (0x4c7b bytes) 000430 1788569891.815747 mapwrite 0x001e53 thru 0x00d835 (0xb9e3 bytes) 000431 1788569891.911512 read 0x021e2b thru 0x0274ed (0x56c3 bytes) 000432 1788569891.912212 trunc from 0x040000 to 0x0168e6 (0x29719 bytes) 000433 1788569891.948117 read 0x0028eb thru 0x00821e (0x5934 bytes) 000434 1788569891.954540 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x168e7): Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x168e7): Mirror 11: 0x74cfae74 (stale) Mirror 13: 0x74cfae74 (stale) Mirror 14: 0x9120695c (stale) CRC-32 checksum value for chunk [0, 0x168e7): Mirror 1: 0x16b62fda Mirror 15: 0x16b62fda lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x168e7: skipped 000435 1788569893.174050 mapwrite 0x038123 thru 0x03ac58 (0x2b36 bytes) 000436 1788569893.410175 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3ac59): Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ac59): Mirror 11: 0xd88f94b1 (stale) Mirror 13: 0xd88f94b1 (stale) Mirror 14: 0x9d0cabf2 (stale) Mirror 15: 0x98863cc9 (stale) CRC-32 checksum value for chunk [0, 0x3ac59): Mirror 1: 0xe65fddb Mirror 16: 0xe65fddb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ac59: skipped 000438 1788569894.433788 mapread 0x03ab84 thru 0x03ac58 (0xd5 bytes) 000441 1788569894.442070 mapread 0x00bb70 thru 0x01591d (0x9dae bytes) 000444 1788569894.471141 write 0x03e67e thru 0x03ffff (0x1982 bytes) 000445 1788569894.632949 punch from 0x00fcbc to 0x019433 (0x9778 bytes) 000448 1788569894.649201 trunc from 0x040000 to 0x002ca5 (0x3d35a bytes) 000449 1788569894.716658 mapwrite 0x00edb7 thru 0x01481d (0x5a67 bytes) 000450 1788569894.831320 punch from 0x00704e to 0x00c450 (0x5403 bytes) 000451 1788569894.844106 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1481e): Mirror 11: 0x95ae4764 (stale) Mirror 13: 0x95ae4764 (stale) Mirror 15: 0x24d358c0 (stale) Mirror 16: 0x24d358c0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1481e): Mirror 11: 0x95ae4764 (stale) Mirror 13: 0x95ae4764 (stale) Mirror 15: 0x24d358c0 (stale) Mirror 16: 0x24d358c0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1481e: skipped 000452 1788569895.378952 trunc from 0x01481e to 0x03e080 (0x29863 bytes) 000453 1788569895.404556 read 0x017a46 thru 0x0195c2 (0x1b7d bytes) 000454 1788569895.417682 mapwrite 0x036006 thru 0x03e695 (0x8690 bytes) 000455 1788569895.553023 write 0x000dee thru 0x004142 (0x3355 bytes) 000456 1788569895.572815 punch from 0x029113 to 0x03743c (0xe32a bytes) 000457 1788569895.594957 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3e696): Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e696): Mirror 11: 0x44967a46 (stale) Mirror 13: 0x44967a46 (stale) Mirror 15: 0x2febacd4 (stale) Mirror 16: 0xc13cdbab (stale) CRC-32 checksum value for chunk [0, 0x3e696): Mirror 1: 0x7aed0560 Mirror 17: 0x7aed0560 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e696: skipped 000458 1788569896.685532 read 0x02cffb thru 0x03c937 (0xf93d bytes) 000459 1788569896.702995 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e696): Mirror 11: 0x44967a46 (stale) Mirror 15: 0x2febacd4 (stale) Mirror 16: 0xc13cdbab (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e696): Mirror 11: 0x44967a46 (stale) Mirror 15: 0x2febacd4 (stale) Mirror 16: 0xc13cdbab (stale) CRC-32 checksum value for chunk [0, 0x3e696): Mirror 1: 0x7aed0560 Mirror 17: 0x7aed0560 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e696: skipped 000460 1788569897.531088 read 0x021521 thru 0x029e97 (0x8977 bytes) 000461 1788569897.562997 read 0x01bf0c thru 0x029130 (0xd225 bytes) 000462 1788569897.588089 mapread 0x03e66a thru 0x03e695 (0x2c bytes) 000464 1788569897.624166 punch from 0x020dd3 to 0x02d07f (0xc2ad bytes) 000465 1788569897.713570 write 0x02827e thru 0x035eed (0xdc70 bytes) 000466 1788569897.725022 trunc from 0x03e696 to 0x03897c (0x5d19 bytes) 000467 1788569897.755416 mapwrite 0x013c21 thru 0x016c18 (0x2ff8 bytes) 000469 1788569897.842208 mapread 0x028cb5 thru 0x02f685 (0x69d1 bytes) 000470 1788569897.844550 write 0x029ee6 thru 0x03647b (0xc596 bytes) 000471 1788569897.855701 read 0x030b28 thru 0x034914 (0x3ded bytes) 000473 1788569897.857422 mapwrite 0x03df77 thru 0x03ffff (0x2089 bytes) 000474 1788569897.943490 mapwrite 0x033835 thru 0x03a764 (0x6f30 bytes) 000475 1788569897.997797 mapwrite 0x002d17 thru 0x011686 (0xe970 bytes) 000476 1788569898.051805 write 0x039bd6 thru 0x03eb55 (0x4f80 bytes) 000477 1788569898.058988 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0x29bbff30 (stale) Mirror 16: 0x7049d4f0 (stale) Mirror 17: 0x5809798 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0x29bbff30 (stale) Mirror 16: 0x7049d4f0 (stale) Mirror 17: 0x5809798 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000478 1788569898.408668 mapread 0x018909 thru 0x01f075 (0x676d bytes) 000479 1788569898.438063 MIRROR_RESYNC 4 mirrors 000481 1788569900.125964 MIRROR_EXTEND to 5 mirrors 000482 1788569901.106100 mapwrite 0x007b79 thru 0x00af30 (0x33b8 bytes) 000486 1788569901.516650 trunc from 0x040000 to 0x0241ce (0x1be31 bytes) 000488 1788569901.602610 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x241cf): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x241cf): Mirror 15: 0x50812d9 (stale) Mirror 16: 0x50812d9 (stale) Mirror 17: 0x50812d9 (stale) Mirror 18: 0x50812d9 (stale) CRC-32 checksum value for chunk [0, 0x241cf): Mirror 1: 0xd8bdeb06 Mirror 19: 0xd8bdeb06 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x241cf: skipped 000489 1788569903.551102 trunc from 0x0241cf to 0x0252d7 (0x1109 bytes) 000490 1788569903.711958 mapread 0x018f0b thru 0x021a9d (0x8b93 bytes) 000491 1788569903.734694 read 0x004999 thru 0x00e647 (0x9caf bytes) 000492 1788569903.743616 punch from 0x009c9d to 0x01818a (0xe4ee bytes) 000494 1788569903.750219 mapwrite 0x019745 thru 0x027a32 (0xe2ee bytes) 000495 1788569903.846681 read 0x01de60 thru 0x027a32 (0x9bd3 bytes) 000496 1788569903.848076 mapwrite 0x02e782 thru 0x036932 (0x81b1 bytes) 000497 1788569903.942709 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36933): Mirror 15: 0xc4bfbccc (stale) Mirror 16: 0xc4bfbccc (stale) Mirror 17: 0xc4bfbccc (stale) Mirror 18: 0xc4bfbccc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36933): Mirror 15: 0xc4bfbccc (stale) Mirror 16: 0xc4bfbccc (stale) Mirror 17: 0xc4bfbccc (stale) Mirror 18: 0xc4bfbccc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36933: skipped 000498 1788569904.547817 read 0x005792 thru 0x005963 (0x1d2 bytes) 000501 1788569904.562823 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36933): Mirror 15: 0xc4bfbccc (stale) Mirror 17: 0xc4bfbccc (stale) Mirror 18: 0xc4bfbccc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36933): Mirror 15: 0xc4bfbccc (stale) Mirror 17: 0xc4bfbccc (stale) Mirror 18: 0xc4bfbccc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36933: skipped 000504 1788569905.002302 mapread 0x0307a9 thru 0x030a5e (0x2b6 bytes) 000505 1788569905.011902 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x36933): Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36933): Mirror 15: 0xc4bfbccc (stale) Mirror 17: 0xc4bfbccc (stale) Mirror 18: 0xc4bfbccc (stale) CRC-32 checksum value for chunk [0, 0x36933): Mirror 1: 0x74157da9 Mirror 19: 0x74157da9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36933: skipped 000506 1788569906.268744 trunc from 0x036933 to 0x013fc8 (0x2296a bytes) 000507 1788569906.367364 trunc from 0x013fc9 to 0x0168e3 (0x291b bytes) 000508 1788569906.388644 mapread 0x00c82c thru 0x0168e3 (0xa0b8 bytes) 000510 1788569906.412528 mapwrite 0x03c6a0 thru 0x03ffff (0x3960 bytes) 000511 1788569906.485249 write 0x03b785 thru 0x03ffff (0x487b bytes) 000512 1788569906.486284 read 0x035b39 thru 0x03c272 (0x673a bytes) 000513 1788569906.486883 write 0x03a469 thru 0x03ffff (0x5b97 bytes) 000516 1788569906.494802 MIRROR_RESYNC 5 mirrors 000517 1788569908.200367 trunc from 0x040000 to 0x03b4e7 (0x4b18 bytes) 000518 1788569908.289491 punch from 0x003345 to 0x00dbee (0xa8aa bytes) 000519 1788569908.296851 trunc from 0x03b4e8 to 0x03aee8 (0x5ff bytes) 000521 1788569908.321729 read 0x02ba69 thru 0x03aa8a (0xf022 bytes) 000523 1788569908.332203 write 0x0319c3 thru 0x03252c (0xb6a bytes) 000525 1788569908.333585 mapread 0x02f664 thru 0x03aee8 (0xb885 bytes) 000526 1788569908.343391 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3aee9): Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3aee9): Mirror 15: 0x6bf6f274 (stale) Mirror 17: 0x6bf6f274 (stale) Mirror 18: 0x6bf6f274 (stale) Mirror 19: 0x6bf6f274 (stale) CRC-32 checksum value for chunk [0, 0x3aee9): Mirror 1: 0xc12392de Mirror 20: 0xc12392de lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3aee9: skipped 000528 1788569910.054940 punch from 0x038d18 to 0x03aee8 (0x21d1 bytes) 000532 1788569910.240163 mapread 0x028ce4 thru 0x030174 (0x7491 bytes) 000533 1788569910.265914 trunc from 0x03aee9 to 0x03a16d (0xd7b bytes) 000534 1788569910.305223 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3a16e): Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a16e): Mirror 15: 0x34e1e87b (stale) Mirror 17: 0x34e1e87b (stale) Mirror 18: 0x34e1e87b (stale) Mirror 19: 0x34e1e87b (stale) Mirror 20: 0x276a74e0 (stale) CRC-32 checksum value for chunk [0, 0x3a16e): Mirror 1: 0x276a74e0 Mirror 21: 0x276a74e0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a16e: skipped 000535 1788569912.080529 punch from 0x0312da to 0x03a16d (0x8e94 bytes) 000536 1788569912.191100 mapwrite 0x036899 thru 0x03729f (0xa07 bytes) 000537 1788569912.259170 punch from 0x024bbe to 0x02e4fb (0x993e bytes) 000539 1788569912.272652 trunc from 0x03a16e to 0x03b8af (0x1742 bytes) 000540 1788569912.295370 punch from 0x026e8b to 0x02705a (0x1d0 bytes) 000541 1788569912.300929 punch from 0x026fb9 to 0x035d8f (0xedd7 bytes) 000542 1788569912.306177 mapwrite 0x0178ae thru 0x023271 (0xb9c4 bytes) 000543 1788569912.439366 mapwrite 0x0332ad thru 0x03a4e1 (0x7235 bytes) 000544 1788569912.576209 write 0x0022f6 thru 0x003655 (0x1360 bytes) 000546 1788569912.594843 write 0x022615 thru 0x02d874 (0xb260 bytes) 000547 1788569912.596442 punch from 0x0004ac to 0x008ccf (0x8824 bytes) 000548 1788569912.613625 trunc from 0x03b8b0 to 0x02949f (0x12410 bytes) 000549 1788569912.634446 mapwrite 0x0320e2 thru 0x03fb57 (0xda76 bytes) 000550 1788569912.767893 read 0x00f8ee thru 0x013733 (0x3e46 bytes) 000552 1788569912.778940 trunc from 0x03fb58 to 0x00d5fb (0x3255c bytes) 000553 1788569912.803395 mapread 0x00a398 thru 0x00d5fb (0x3264 bytes) 000556 1788569912.820394 punch from 0x00d129 to 0x00d5fb (0x4d3 bytes) 000557 1788569912.831714 read 0x008fd8 thru 0x00d5fb (0x4624 bytes) 000558 1788569912.845836 trunc from 0x00d5fc to 0x023e1a (0x1681f bytes) 000560 1788569912.877738 trunc from 0x023e1b to 0x00feb6 (0x13f64 bytes) 000561 1788569912.911492 mapwrite 0x03d198 thru 0x03ffff (0x2e68 bytes) 000562 1788569913.065637 trunc from 0x040000 to 0x00ae0f (0x351f0 bytes) 000564 1788569913.090771 punch from 0x009f81 to 0x00ae0f (0xe8f bytes) 000565 1788569913.097468 write 0x0245ca thru 0x02609c (0x1ad3 bytes) 000566 1788569913.099104 mapread 0x01caf5 thru 0x02609c (0x95a8 bytes) 000567 1788569913.138722 punch from 0x018cc9 to 0x01e9ba (0x5cf2 bytes) 000568 1788569913.151868 mapwrite 0x025fe9 thru 0x0354b1 (0xf4c9 bytes) 000569 1788569913.382508 write 0x038930 thru 0x03a4e2 (0x1bb3 bytes) 000571 1788569913.385024 write 0x03805e thru 0x03ade8 (0x2d8b bytes) 000573 1788569913.386357 mapwrite 0x03c7c6 thru 0x03ffff (0x383a bytes) 000574 1788569913.485556 mapwrite 0x026c2a thru 0x035158 (0xe52f bytes) 000576 1788569913.581193 punch from 0x00cee1 to 0x019ec9 (0xcfe9 bytes) 000580 1788569913.590637 punch from 0x019768 to 0x026896 (0xd12f bytes) 000581 1788569913.603925 punch from 0x001e3d to 0x00c268 (0xa42c bytes) 000583 1788569913.615923 write 0x02c5b8 thru 0x02df15 (0x195e bytes) 000584 1788569913.617422 mapread 0x022ea3 thru 0x027391 (0x44ef bytes) 000585 1788569913.656432 punch from 0x0157d7 to 0x0245cc (0xedf6 bytes) 000588 1788569913.671362 mapread 0x03abbc thru 0x03ffff (0x5444 bytes) 000589 1788569913.688674 read 0x005d87 thru 0x01085d (0xaad7 bytes) 000590 1788569913.715198 read 0x03bac7 thru 0x03ffff (0x4539 bytes) 000592 1788569913.716328 punch from 0x02ba82 to 0x038bd0 (0xd14f bytes) 000593 1788569913.725878 mapwrite 0x03fc34 thru 0x03ffff (0x3cc bytes) 000599 1788569913.863928 mapwrite 0x0109ac thru 0x01f81a (0xee6f bytes) 000600 1788569914.022551 punch from 0x01f727 to 0x021432 (0x1d0c bytes) 000602 1788569914.047648 punch from 0x024c3d to 0x02829f (0x3663 bytes) 000603 1788569914.073163 MIRROR_RESYNC 7 mirrors 000604 1788569920.051971 MIRROR_EXTEND to 8 mirrors 000605 1788569921.291165 read 0x038aaa thru 0x03f0aa (0x6601 bytes) 000606 1788569921.300201 mapwrite 0x017442 thru 0x024ffa (0xdbb9 bytes) 000607 1788569921.585355 read 0x00582c thru 0x009268 (0x3a3d bytes) 000608 1788569921.608631 write 0x023f3e thru 0x02ef06 (0xafc9 bytes) 000609 1788569921.617197 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0xc7406d84 (stale) Mirror 18: 0xc7406d84 (stale) Mirror 19: 0xc7406d84 (stale) Mirror 20: 0xc7406d84 (stale) Mirror 21: 0xc7406d84 (stale) Mirror 22: 0xc7406d84 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0xc7406d84 (stale) Mirror 18: 0xc7406d84 (stale) Mirror 19: 0xc7406d84 (stale) Mirror 20: 0xc7406d84 (stale) Mirror 21: 0xc7406d84 (stale) Mirror 22: 0xc7406d84 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000610 1788569922.410728 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0xc7406d84 (stale) Mirror 18: 0xc7406d84 (stale) Mirror 19: 0xc7406d84 (stale) Mirror 20: 0xc7406d84 (stale) Mirror 21: 0xc7406d84 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0xc7406d84 (stale) Mirror 18: 0xc7406d84 (stale) Mirror 19: 0xc7406d84 (stale) Mirror 20: 0xc7406d84 (stale) Mirror 21: 0xc7406d84 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000611 1788569923.076309 trunc from 0x040000 to 0x03bd6d (0x4292 bytes) 000612 1788569923.109515 trunc from 0x03bd6e to 0x008967 (0x33406 bytes) 000614 1788569923.137979 write 0x01643b thru 0x02501f (0xebe5 bytes) 000615 1788569923.151749 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x25020): Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25020): Mirror 17: 0x95828dd9 (stale) Mirror 18: 0x95828dd9 (stale) Mirror 19: 0x95828dd9 (stale) Mirror 20: 0x95828dd9 (stale) Mirror 21: 0x95828dd9 (stale) CRC-32 checksum value for chunk [0, 0x25020): Mirror 1: 0x37b7539a Mirror 22: 0x37b7539a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25020: skipped 000616 1788569924.484742 mapwrite 0x022489 thru 0x02c6f7 (0xa26f bytes) 000618 1788569924.692240 read 0x016990 thru 0x020258 (0x98c9 bytes) 000619 1788569924.702166 punch from 0x013073 to 0x01ef7b (0xbf09 bytes) 000620 1788569924.707445 mapwrite 0x0141b3 thru 0x0161bb (0x2009 bytes) 000621 1788569924.760466 punch from 0x02aca6 to 0x02c6f7 (0x1a52 bytes) 000622 1788569924.765475 punch from 0x016fa2 to 0x023dc6 (0xce25 bytes) 000623 1788569924.771661 write 0x023277 thru 0x024932 (0x16bc bytes) 000624 1788569924.776694 mapwrite 0x0386c7 thru 0x03e4bb (0x5df5 bytes) 000625 1788569924.862652 MIRROR_RESYNC 7 mirrors 000626 1788569928.753816 read 0x02857f thru 0x036b1d (0xe59f bytes) 000627 1788569928.780505 read 0x016003 thru 0x016b50 (0xb4e bytes) 000628 1788569928.800236 MIRROR_EXTEND to 8 mirrors 000630 1788569930.616873 MIRROR_RESYNC 8 mirrors 000631 1788569931.454774 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) 000633 1788569931.857977 mapread 0x031644 thru 0x035669 (0x4026 bytes) 000634 1788569931.902159 punch from 0x01cae0 to 0x022bc5 (0x60e6 bytes) 000635 1788569932.101329 mapread 0x010335 thru 0x018800 (0x84cc bytes) 000638 1788569932.117465 mapwrite 0x019747 thru 0x01a5f6 (0xeb0 bytes) 000639 1788569932.190551 punch from 0x0372ca to 0x038196 (0xecd bytes) 000640 1788569932.208341 punch from 0x02b79e to 0x032f5a (0x77bd bytes) 000641 1788569932.219807 mapwrite 0x01e432 thru 0x028414 (0x9fe3 bytes) 000642 1788569932.344783 mapread 0x0381cc thru 0x038e93 (0xcc8 bytes) 000646 1788569932.358199 trunc from 0x03e4bc to 0x0061e6 (0x382d5 bytes) 000648 1788569932.401198 mapwrite 0x021a81 thru 0x024106 (0x2686 bytes) 000649 1788569932.561790 punch from 0x020486 to 0x02209d (0x1c18 bytes) 000651 1788569932.569780 mapwrite 0x031b28 thru 0x032226 (0x6ff bytes) 000652 1788569932.709656 write 0x01dd31 thru 0x023496 (0x5766 bytes) 000653 1788569932.728836 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x32227): Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32227): Mirror 17: 0xda288b6d (stale) Mirror 18: 0xda288b6d (stale) Mirror 19: 0xda288b6d (stale) Mirror 20: 0xda288b6d (stale) Mirror 21: 0xda288b6d (stale) Mirror 22: 0xda288b6d (stale) CRC-32 checksum value for chunk [0, 0x32227): Mirror 1: 0x6b855ca5 Mirror 23: 0x6b855ca5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32227: skipped 000655 1788569934.215332 write 0x03af51 thru 0x03ffff (0x50af bytes) 000656 1788569934.334482 trunc from 0x040000 to 0x02028e (0x1fd71 bytes) 000657 1788569934.374038 MIRROR_RESYNC 8 mirrors 000658 1788569936.576491 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) 000659 1788569937.103733 mapread 0x01aadc thru 0x01dc16 (0x313b bytes) 000660 1788569937.134114 MIRROR_EXTEND to 8 mirrors 000661 1788569938.236074 mapread 0x00a531 thru 0x0124d7 (0x7fa7 bytes) 000664 1788569938.273254 trunc from 0x02028f to 0x01d7ee (0x2aa0 bytes) 000668 1788569938.389002 MIRROR_RESYNC 8 mirrors 000669 1788569941.529684 write 0x03cc0e thru 0x03ffff (0x33f2 bytes) 000670 1788569941.710184 trunc from 0x040000 to 0x02d0b0 (0x12f4f bytes) 000671 1788569941.765073 punch from 0x00c9f8 to 0x00d1a7 (0x7b0 bytes) truncating to largest ever: 0x3fd72 000673 1788569941.781259 trunc from 0x02d0b1 to 0x03fd71 (0x12cc1 bytes) 000674 1788569941.843184 read 0x008b8c thru 0x017b19 (0xef8e bytes) 000675 1788569941.877661 mapwrite 0x005515 thru 0x008fc3 (0x3aaf bytes) 000678 1788569942.089177 write 0x00b4eb thru 0x00b913 (0x429 bytes) 000679 1788569942.090047 write 0x03d6f5 thru 0x03ffff (0x290b bytes) 000680 1788569942.117702 punch from 0x0147b6 to 0x01f5f8 (0xae43 bytes) 000681 1788569942.174615 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0xe20eea22 (stale) Mirror 19: 0xe20eea22 (stale) Mirror 20: 0xe20eea22 (stale) Mirror 21: 0xe20eea22 (stale) Mirror 22: 0xe20eea22 (stale) Mirror 23: 0xe20eea22 (stale) Mirror 24: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5e8608c9 Mirror 25: 0x5e8608c9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000682 1788569944.296050 mapread 0x019f0f thru 0x022077 (0x8169 bytes) 000684 1788569944.378291 punch from 0x00f00b to 0x017814 (0x880a bytes) 000685 1788569944.523307 read 0x008355 thru 0x0119ed (0x9699 bytes) 000686 1788569944.543345 punch from 0x01724a to 0x02658f (0xf346 bytes) 000687 1788569944.553059 write 0x016f31 thru 0x022697 (0xb767 bytes) 000690 1788569944.573654 mapwrite 0x03f811 thru 0x03ffff (0x7ef bytes) 000692 1788569944.752535 write 0x015e26 thru 0x01889e (0x2a79 bytes) 000693 1788569944.767200 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0xe20eea22 (stale) Mirror 20: 0xe20eea22 (stale) Mirror 21: 0xe20eea22 (stale) Mirror 22: 0xe20eea22 (stale) Mirror 23: 0xe20eea22 (stale) Mirror 24: 0xe20eea22 (stale) Mirror 25: 0x5e8608c9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0xe20eea22 (stale) Mirror 20: 0xe20eea22 (stale) Mirror 21: 0xe20eea22 (stale) Mirror 22: 0xe20eea22 (stale) Mirror 23: 0xe20eea22 (stale) Mirror 24: 0xe20eea22 (stale) Mirror 25: 0x5e8608c9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000694 1788569945.547083 mapread 0x03bfd0 thru 0x03ffff (0x4030 bytes) 000695 1788569945.567033 mapread 0x00445d thru 0x009658 (0x51fc bytes) 000696 1788569945.580790 MIRROR_RESYNC 8 mirrors 000697 1788569952.952973 read 0x026e11 thru 0x030f49 (0xa139 bytes) 000698 1788569952.964899 trunc from 0x040000 to 0x015a61 (0x2a59e bytes) 000703 1788569953.059229 mapread 0x007a68 thru 0x00e7e7 (0x6d80 bytes) 000705 1788569953.081957 read 0x005f43 thru 0x00b3d9 (0x5497 bytes) 000706 1788569953.082840 trunc from 0x015a62 to 0x03e931 (0x28ed0 bytes) 000708 1788569953.111010 punch from 0x013c52 to 0x01c4dc (0x888b bytes) 000709 1788569953.117399 write 0x01cfe2 thru 0x024507 (0x7526 bytes) 000710 1788569953.132423 mapwrite 0x013cbb thru 0x01f858 (0xbb9e bytes) 000711 1788569953.210863 trunc from 0x03e932 to 0x024a1a (0x19f17 bytes) 000712 1788569953.234650 read 0x00bf3e thru 0x01ab85 (0xec48 bytes) 000713 1788569953.235435 punch from 0x008084 to 0x01118c (0x9109 bytes) 000715 1788569953.248034 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24a1b): Mirror 20: 0xa86c8c83 (stale) Mirror 21: 0xa86c8c83 (stale) Mirror 22: 0xa86c8c83 (stale) Mirror 23: 0xa86c8c83 (stale) Mirror 24: 0xa86c8c83 (stale) Mirror 25: 0xa86c8c83 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24a1b): Mirror 20: 0xa86c8c83 (stale) Mirror 21: 0xa86c8c83 (stale) Mirror 22: 0xa86c8c83 (stale) Mirror 23: 0xa86c8c83 (stale) Mirror 24: 0xa86c8c83 (stale) Mirror 25: 0xa86c8c83 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24a1b: skipped 000717 1788569954.069699 mapwrite 0x02e7f0 thru 0x03969d (0xaeae bytes) 000718 1788569954.315020 mapwrite 0x038ea6 thru 0x03ef90 (0x60eb bytes) 000719 1788569954.526969 mapread 0x01f637 thru 0x027603 (0x7fcd bytes) 000720 1788569954.564129 mapread 0x015d43 thru 0x023bd3 (0xde91 bytes) 000721 1788569954.598829 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ef91): Mirror 20: 0xf8b15913 (stale) Mirror 21: 0xf8b15913 (stale) Mirror 22: 0xf8b15913 (stale) Mirror 23: 0xf8b15913 (stale) Mirror 25: 0xf8b15913 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ef91): Mirror 20: 0xf8b15913 (stale) Mirror 21: 0xf8b15913 (stale) Mirror 22: 0xf8b15913 (stale) Mirror 23: 0xf8b15913 (stale) Mirror 25: 0xf8b15913 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ef91: skipped 000722 1788569955.344119 trunc from 0x03ef91 to 0x01d044 (0x21f4c bytes) 000723 1788569955.389137 punch from 0x016da8 to 0x01adbb (0x4014 bytes) 000726 1788569955.402148 trunc from 0x01d045 to 0x013232 (0x9e12 bytes) 000727 1788569955.426545 write 0x00ddc8 thru 0x01b6f4 (0xd92d bytes) 000728 1788569955.433734 read 0x0113a9 thru 0x01b6f4 (0xa34c bytes) 000729 1788569955.434431 read 0x004961 thru 0x0146a4 (0xfd44 bytes) 000730 1788569955.443277 mapread 0x01a904 thru 0x01b6f4 (0xdf1 bytes) 000731 1788569955.444402 mapwrite 0x0080eb thru 0x010a20 (0x8936 bytes) 000734 1788569955.497097 write 0x03bc6e thru 0x03ffff (0x4392 bytes) 000735 1788569955.498231 trunc from 0x040000 to 0x005536 (0x3aac9 bytes) 000736 1788569955.516129 mapread 0x0052fd thru 0x005536 (0x23a bytes) 000738 1788569955.518361 mapread 0x003ed1 thru 0x005536 (0x1666 bytes) 000741 1788569955.522990 trunc from 0x005537 to 0x004aac (0xa8a bytes) 000743 1788569955.544667 write 0x03ac2f thru 0x03c60d (0x19df bytes) 000745 1788569955.547347 write 0x01a654 thru 0x02534a (0xacf7 bytes) 000746 1788569955.555749 write 0x026e33 thru 0x02e35f (0x752d bytes) 000747 1788569955.557715 mapwrite 0x009dd0 thru 0x00b9ba (0x1beb bytes) 000748 1788569955.681027 trunc from 0x03c60e to 0x035eb3 (0x675a bytes) 000749 1788569955.729089 punch from 0x014ce0 to 0x017a78 (0x2d99 bytes) 000750 1788569955.753770 punch from 0x0060ba to 0x014793 (0xe6da bytes) 000751 1788569955.761953 mapread 0x00b466 thru 0x01994a (0xe4e5 bytes) 000752 1788569955.863145 mapwrite 0x015cdf thru 0x01dad2 (0x7df4 bytes) 000753 1788569955.935978 mapwrite 0x00d155 thru 0x010e59 (0x3d05 bytes) 000754 1788569955.998001 punch from 0x007e6c to 0x013ff4 (0xc189 bytes) 000755 1788569956.016542 write 0x03b260 thru 0x03ffff (0x4da0 bytes) 000756 1788569956.021614 punch from 0x032018 to 0x0373f0 (0x53d9 bytes) 000757 1788569956.051913 write 0x01a533 thru 0x01a912 (0x3e0 bytes) 000758 1788569956.060510 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: 0xc124fd7f (stale) Mirror 21: 0xc124fd7f (stale) Mirror 22: 0xc124fd7f (stale) Mirror 23: 0xc124fd7f (stale) Mirror 25: 0xc124fd7f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7905f821 Mirror 26: 0x7905f821 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000760 1788569958.273650 trunc from 0x040000 to 0x003aae (0x3c551 bytes) 000761 1788569958.513621 write 0x02deb8 thru 0x02f6f9 (0x1842 bytes) 000764 1788569958.517799 mapwrite 0x00acfd thru 0x014180 (0x9484 bytes) 000765 1788569958.750442 mapwrite 0x017c4d thru 0x01ef96 (0x734a bytes) 000769 1788569959.032305 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2f6fa): Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f6fa): Mirror 20: 0xde49406f (stale) Mirror 21: 0xde49406f (stale) Mirror 22: 0xde49406f (stale) Mirror 23: 0xde49406f (stale) Mirror 25: 0xde49406f (stale) Mirror 26: 0xe6e65efd (stale) CRC-32 checksum value for chunk [0, 0x2f6fa): Mirror 1: 0xdf451df Mirror 27: 0xdf451df lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f6fa: skipped 000770 1788569961.277041 mapread 0x001a14 thru 0x009829 (0x7e16 bytes) 000771 1788569961.355619 trunc from 0x02f6fa to 0x00e6ab (0x2104e bytes) 000772 1788569961.658339 mapwrite 0x03d98c thru 0x03ffff (0x2674 bytes) 000773 1788569961.797816 mapwrite 0x03b95d thru 0x03ffff (0x46a3 bytes) 000775 1788569961.856944 read 0x002747 thru 0x004006 (0x18c0 bytes) 000776 1788569961.863086 punch from 0x030ae0 to 0x035dc9 (0x52ea bytes) 000779 1788569961.869047 write 0x02a551 thru 0x035a5f (0xb50f bytes) 000780 1788569961.878770 read 0x019d17 thru 0x01ce2b (0x3115 bytes) 000782 1788569961.891229 read 0x00603f thru 0x00d7fa (0x77bc bytes) 000783 1788569961.893892 write 0x025b90 thru 0x029c1f (0x4090 bytes) 000787 1788569961.896527 write 0x008e9b thru 0x016c44 (0xddaa bytes) 000788 1788569961.898385 read 0x02203e thru 0x024d7b (0x2d3e bytes) 000789 1788569961.899267 read 0x03ede3 thru 0x03ffff (0x121d bytes) 000790 1788569961.900029 read 0x01c51a thru 0x0225f7 (0x60de bytes) 000792 1788569961.901502 write 0x01cf1d thru 0x023d9b (0x6e7f bytes) 000793 1788569961.902964 mapread 0x00d2e1 thru 0x00eb80 (0x18a0 bytes) 000794 1788569961.904542 punch from 0x01fb7b to 0x029e62 (0xa2e8 bytes) 000795 1788569961.932291 mapwrite 0x01de42 thru 0x023ab3 (0x5c72 bytes) 000796 1788569962.016269 read 0x035d9a thru 0x037d2c (0x1f93 bytes) 000798 1788569962.017308 punch from 0x026ff7 to 0x030a87 (0x9a91 bytes) 000799 1788569962.027476 trunc from 0x040000 to 0x013a0e (0x2c5f1 bytes) 000801 1788569962.076045 read 0x013252 thru 0x013a0e (0x7bd bytes) 000802 1788569962.077142 mapread 0x0037e3 thru 0x009c88 (0x64a6 bytes) 000803 1788569962.078217 punch from 0x008d95 to 0x012adc (0x9d48 bytes) 000804 1788569962.083597 punch from 0x00f792 to 0x013a0e (0x427d bytes) 000805 1788569962.094969 write 0x01e60b thru 0x01ef3a (0x930 bytes) 000806 1788569962.104630 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x1ef3b): Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ef3b): Mirror 20: 0x67bd865e (stale) Mirror 21: 0x67bd865e (stale) Mirror 22: 0x67bd865e (stale) Mirror 23: 0x67bd865e (stale) Mirror 25: 0x67bd865e (stale) Mirror 26: 0x18200d3c (stale) Mirror 27: 0x9fa9057d (stale) CRC-32 checksum value for chunk [0, 0x1ef3b): Mirror 1: 0x35d3fdd Mirror 28: 0x35d3fdd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ef3b: skipped 000808 1788569963.684488 punch from 0x00b3c1 to 0x00e366 (0x2fa6 bytes) 000809 1788569963.828187 mapread 0x005209 thru 0x00c412 (0x720a bytes) 000810 1788569963.910283 trunc from 0x01ef3b to 0x010615 (0xe925 bytes) 000811 1788569964.002726 punch from 0x00f9b1 to 0x010615 (0xc65 bytes) 000813 1788569964.020399 punch from 0x0080b7 to 0x010615 (0x855f bytes) 000814 1788569964.056219 MIRROR_RESYNC 9 mirrors 000818 1788569967.936856 punch from 0x008ae3 to 0x010615 (0x7b33 bytes) 000819 1788569968.014386 mapwrite 0x0003ff thru 0x003f1b (0x3b1d bytes) 000820 1788569968.202378 mapwrite 0x02ca1c thru 0x036a48 (0xa02d bytes) 000821 1788569968.462953 mapread 0x012a75 thru 0x013cd6 (0x1262 bytes) 000822 1788569968.492842 punch from 0x00339e to 0x00a9fa (0x765d bytes) 000826 1788569968.518106 trunc from 0x036a49 to 0x00e692 (0x283b6 bytes) 000827 1788569968.575161 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xe693): Mirror 20: 0xcaaaa1e8 (stale) Mirror 21: 0xcaaaa1e8 (stale) Mirror 22: 0xcaaaa1e8 (stale) Mirror 23: 0xcaaaa1e8 (stale) Mirror 25: 0xcaaaa1e8 (stale) Mirror 27: 0xcaaaa1e8 (stale) Mirror 28: 0xcaaaa1e8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe693): Mirror 20: 0xcaaaa1e8 (stale) Mirror 21: 0xcaaaa1e8 (stale) Mirror 22: 0xcaaaa1e8 (stale) Mirror 23: 0xcaaaa1e8 (stale) Mirror 25: 0xcaaaa1e8 (stale) Mirror 27: 0xcaaaa1e8 (stale) Mirror 28: 0xcaaaa1e8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe693: skipped 000828 1788569969.481947 write 0x03cea7 thru 0x03ffff (0x3159 bytes) 000829 1788569969.483017 punch from 0x00f613 to 0x015784 (0x6172 bytes) 000831 1788569969.495717 write 0x01e9a7 thru 0x02c42c (0xda86 bytes) 000832 1788569969.517168 read 0x025b2c thru 0x03454c (0xea21 bytes) 000834 1788569969.538531 punch from 0x02b476 to 0x03752f (0xc0ba bytes) 000836 1788569969.589267 MIRROR_RESYNC 8 mirrors 000837 1788569973.791498 trunc from 0x040000 to 0x013660 (0x2c99f bytes) 000838 1788569973.922795 mapwrite 0x03cfc8 thru 0x03ffff (0x3038 bytes) 000839 1788569974.076829 punch from 0x022f45 to 0x02b415 (0x84d1 bytes) 000840 1788569974.088207 mapwrite 0x002a4f thru 0x00a8e5 (0x7e97 bytes) 000841 1788569974.148864 punch from 0x00a010 to 0x00e5e2 (0x45d3 bytes) 000843 1788569974.164402 mapread 0x0063c3 thru 0x00dfab (0x7be9 bytes) 000844 1788569974.187499 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: 0x408a0e50 (stale) Mirror 21: 0x408a0e50 (stale) Mirror 22: 0x408a0e50 (stale) Mirror 23: 0x408a0e50 (stale) Mirror 25: 0x408a0e50 (stale) Mirror 28: 0x408a0e50 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: 0x408a0e50 (stale) Mirror 21: 0x408a0e50 (stale) Mirror 22: 0x408a0e50 (stale) Mirror 23: 0x408a0e50 (stale) Mirror 25: 0x408a0e50 (stale) Mirror 28: 0x408a0e50 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000845 1788569974.854434 MIRROR_RESYNC 7 mirrors 000846 1788569977.725197 read 0x01640e thru 0x01e1cb (0x7dbe bytes) 000847 1788569977.740493 trunc from 0x040000 to 0x02ba54 (0x145ab bytes) 000848 1788569977.890865 read 0x0089de thru 0x00dc23 (0x5246 bytes) 000849 1788569977.910443 read 0x017310 thru 0x022494 (0xb185 bytes) 000850 1788569977.915294 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2ba55): Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ba55): Mirror 20: 0xcc83609e (stale) Mirror 21: 0xcc83609e (stale) Mirror 22: 0xcc83609e (stale) Mirror 23: 0xcc83609e (stale) Mirror 25: 0xcc83609e (stale) Mirror 28: 0xcc83609e (stale) CRC-32 checksum value for chunk [0, 0x2ba55): Mirror 1: 0xcc83609e Mirror 29: 0xcc83609e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ba55: skipped 000851 1788569979.803545 MIRROR_RESYNC 8 mirrors 000852 1788569982.157649 punch from 0x00a14c to 0x010f6f (0x6e24 bytes) 000853 1788569982.258834 mapwrite 0x00950e thru 0x00cd4e (0x3841 bytes) 000854 1788569982.348792 punch from 0x00adec to 0x013ba1 (0x8db6 bytes) 000855 1788569982.357523 mapread 0x000806 thru 0x00fe0f (0xf60a bytes) 000856 1788569982.371339 mapread 0x01d5d9 thru 0x022bbb (0x55e3 bytes) 000857 1788569982.385231 punch from 0x00822d to 0x00e9a3 (0x6777 bytes) 000858 1788569982.392997 mapwrite 0x03effb thru 0x03ffff (0x1005 bytes) 000859 1788569982.487214 write 0x00d708 thru 0x00f66d (0x1f66 bytes) 000860 1788569982.496832 read 0x0101bf thru 0x012c0b (0x2a4d bytes) 000861 1788569982.502823 read 0x011d29 thru 0x0155e6 (0x38be bytes) 000862 1788569982.523435 mapwrite 0x02e72a thru 0x03affd (0xc8d4 bytes) 000864 1788569982.629953 trunc from 0x040000 to 0x003b7e (0x3c481 bytes) 000865 1788569982.650391 mapread 0x002125 thru 0x003b7e (0x1a5a bytes) 000866 1788569982.652327 punch from 0x00006b to 0x003b7e (0x3b14 bytes) 000867 1788569982.664473 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3b7f): Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b7f): Mirror 20: 0x25358e1a (stale) Mirror 21: 0x25358e1a (stale) Mirror 22: 0x25358e1a (stale) Mirror 23: 0x25358e1a (stale) Mirror 25: 0x25358e1a (stale) Mirror 28: 0x25358e1a (stale) Mirror 29: 0x25358e1a (stale) CRC-32 checksum value for chunk [0, 0x3b7f): Mirror 1: 0x10e99e36 Mirror 30: 0x10e99e36 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b7f: skipped 000868 1788569984.311484 read 0x000150 thru 0x003b7e (0x3a2f bytes) 000869 1788569984.330760 read 0x0038b3 thru 0x003b7e (0x2cc bytes) 000871 1788569984.354858 write 0x000466 thru 0x009c63 (0x97fe bytes) 000872 1788569984.527235 mapwrite 0x030dff thru 0x03ca4f (0xbc51 bytes) 000873 1788569984.723964 read 0x02c8cb thru 0x034ade (0x8214 bytes) 000874 1788569984.725285 mapread 0x00b0ac thru 0x00dd70 (0x2cc5 bytes) 000875 1788569984.729332 punch from 0x01f20c to 0x024892 (0x5687 bytes) 000876 1788569984.737836 mapwrite 0x006df1 thru 0x011790 (0xa9a0 bytes) 000877 1788569984.831407 read 0x00cdbd thru 0x014bca (0x7e0e bytes) 000878 1788569984.832042 mapwrite 0x013278 thru 0x020a96 (0xd81f bytes) 000879 1788569984.935154 trunc from 0x03ca50 to 0x000bbf (0x3be90 bytes) 000880 1788569984.975420 write 0x016f2e thru 0x021383 (0xa456 bytes) 000881 1788569984.979217 mapread 0x0081e7 thru 0x00be41 (0x3c5b bytes) 000883 1788569985.020606 mapread 0x018f76 thru 0x01cd1c (0x3da7 bytes) 000885 1788569985.032415 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21384): Mirror 21: 0x3670e6a9 (stale) Mirror 22: 0x3670e6a9 (stale) Mirror 23: 0x3670e6a9 (stale) Mirror 25: 0x3670e6a9 (stale) Mirror 28: 0x3670e6a9 (stale) Mirror 29: 0x3670e6a9 (stale) Mirror 30: 0xcb51a756 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21384): Mirror 21: 0x3670e6a9 (stale) Mirror 22: 0x3670e6a9 (stale) Mirror 23: 0x3670e6a9 (stale) Mirror 25: 0x3670e6a9 (stale) Mirror 28: 0x3670e6a9 (stale) Mirror 29: 0x3670e6a9 (stale) Mirror 30: 0xcb51a756 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21384: skipped 000888 1788569985.738737 write 0x00dc46 thru 0x019bf1 (0xbfac bytes) 000890 1788569985.763317 punch from 0x0046bd to 0x010538 (0xbe7c bytes) 000891 1788569985.791502 mapwrite 0x0376c1 thru 0x03ffff (0x893f bytes) 000892 1788569985.923001 mapwrite 0x010cfc thru 0x01d657 (0xc95c bytes) 000894 1788569986.039045 trunc from 0x040000 to 0x035b4d (0xa4b2 bytes) 000895 1788569986.067364 read 0x02decb thru 0x035b4d (0x7c83 bytes) 000896 1788569986.078414 write 0x03363c thru 0x03ffff (0xc9c4 bytes) 000897 1788569986.079981 write 0x00bd6b thru 0x00e818 (0x2aae bytes) 000899 1788569986.084252 trunc from 0x040000 to 0x035134 (0xaecb bytes) 000900 1788569986.112341 mapwrite 0x039993 thru 0x03ffff (0x666d bytes) 000902 1788569986.205190 punch from 0x03e1b0 to 0x03ffff (0x1e50 bytes) 000903 1788569986.212148 write 0x00b850 thru 0x013394 (0x7b45 bytes) 000904 1788569986.213536 trunc from 0x040000 to 0x0286f1 (0x1790e bytes) 000905 1788569986.240188 mapwrite 0x02d3b3 thru 0x038ec4 (0xbb12 bytes) 000906 1788569986.429537 punch from 0x00b517 to 0x00e608 (0x30f2 bytes) 000908 1788569986.444541 mapwrite 0x023b33 thru 0x0331bf (0xf68d bytes) 000909 1788569986.551791 mapread 0x017378 thru 0x01ffe3 (0x8c6c bytes) 000910 1788569986.553351 mapread 0x021483 thru 0x02b27d (0x9dfb bytes) 000911 1788569986.555220 mapread 0x035b65 thru 0x038ec4 (0x3360 bytes) 000912 1788569986.556707 punch from 0x004d1f to 0x00f25b (0xa53d bytes) 000914 1788569986.571070 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x38ec5): Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38ec5): Mirror 21: 0xa4287956 (stale) Mirror 22: 0xa4287956 (stale) Mirror 23: 0xa4287956 (stale) Mirror 25: 0xa4287956 (stale) Mirror 28: 0xa4287956 (stale) Mirror 29: 0xa4287956 (stale) Mirror 30: 0xa3b0e737 (stale) CRC-32 checksum value for chunk [0, 0x38ec5): Mirror 1: 0x8cf2e89b Mirror 31: 0x8cf2e89b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38ec5: skipped 000915 1788569987.925673 mapwrite 0x038686 thru 0x038af1 (0x46c bytes) 000916 1788569988.202725 write 0x00a73b thru 0x0151e0 (0xaaa6 bytes) 000918 1788569988.214429 trunc from 0x038ec5 to 0x035f36 (0x2f8e bytes) 000919 1788569988.237302 mapwrite 0x011b04 thru 0x021895 (0xfd92 bytes) 000920 1788569988.360467 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35f37): Mirror 21: 0xbedf3623 (stale) Mirror 22: 0xbedf3623 (stale) Mirror 23: 0xbedf3623 (stale) Mirror 28: 0xbedf3623 (stale) Mirror 29: 0xbedf3623 (stale) Mirror 30: 0xb2ba236f (stale) Mirror 31: 0x9454b81f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35f37): Mirror 21: 0xbedf3623 (stale) Mirror 22: 0xbedf3623 (stale) Mirror 23: 0xbedf3623 (stale) Mirror 28: 0xbedf3623 (stale) Mirror 29: 0xbedf3623 (stale) Mirror 30: 0xb2ba236f (stale) Mirror 31: 0x9454b81f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35f37: skipped 000922 1788569989.034485 punch from 0x034b2c to 0x035f36 (0x140b bytes) 000923 1788569989.042077 trunc from 0x035f37 to 0x0183dc (0x1db5a bytes) 000924 1788569989.063685 write 0x017b2a thru 0x01bdee (0x42c5 bytes) 000925 1788569989.071142 mapread 0x00f22b thru 0x019fe1 (0xadb7 bytes) 000926 1788569989.097200 mapwrite 0x01f03c thru 0x027ff1 (0x8fb6 bytes) 000927 1788569989.195231 mapwrite 0x01504c thru 0x01c65f (0x7614 bytes) 000929 1788569989.250487 mapwrite 0x0003b8 thru 0x00b04f (0xac98 bytes) 000930 1788569989.318839 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27ff2): Mirror 21: 0xa548fa92 (stale) Mirror 22: 0xa548fa92 (stale) Mirror 28: 0xa548fa92 (stale) Mirror 29: 0xa548fa92 (stale) Mirror 30: 0xa4b510cc (stale) Mirror 31: 0x6eac9880 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27ff2): Mirror 21: 0xa548fa92 (stale) Mirror 22: 0xa548fa92 (stale) Mirror 28: 0xa548fa92 (stale) Mirror 29: 0xa548fa92 (stale) Mirror 30: 0xa4b510cc (stale) Mirror 31: 0x6eac9880 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27ff2: skipped 000931 1788569989.807658 punch from 0x0165f2 to 0x01df72 (0x7981 bytes) 000933 1788569989.814263 read 0x01b92d thru 0x024347 (0x8a1b bytes) 000934 1788569989.826110 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27ff2): Mirror 21: 0xa548fa92 (stale) Mirror 28: 0xa548fa92 (stale) Mirror 29: 0xa548fa92 (stale) Mirror 30: 0xa4b510cc (stale) Mirror 31: 0x6eac9880 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27ff2): Mirror 21: 0xa548fa92 (stale) Mirror 28: 0xa548fa92 (stale) Mirror 29: 0xa548fa92 (stale) Mirror 30: 0xa4b510cc (stale) Mirror 31: 0x6eac9880 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27ff2: skipped 000937 1788569990.400942 punch from 0x003a20 to 0x01318f (0xf770 bytes) 000938 1788569990.408133 write 0x039bcf thru 0x03ffff (0x6431 bytes) 000939 1788569990.410423 read 0x032c7a thru 0x03e003 (0xb38a bytes) 000940 1788569990.431707 mapread 0x0033ff thru 0x007f94 (0x4b96 bytes) 000943 1788569990.487349 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0xc4550116 (stale) Mirror 29: 0xc4550116 (stale) Mirror 30: 0xe20eea22 (stale) Mirror 31: 0xbead91d1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0xc4550116 (stale) Mirror 29: 0xc4550116 (stale) Mirror 30: 0xe20eea22 (stale) Mirror 31: 0xbead91d1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000944 1788569991.076157 punch from 0x0014e1 to 0x00a77c (0x929c bytes) 000945 1788569991.084152 mapwrite 0x01b2a3 thru 0x023239 (0x7f97 bytes) 000946 1788569991.167747 mapwrite 0x030323 thru 0x033c32 (0x3910 bytes) 000948 1788569991.239209 punch from 0x02e5dc to 0x036250 (0x7c75 bytes) 000951 1788569991.259235 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 29: 0xc4550116 (stale) Mirror 30: 0xe20eea22 (stale) Mirror 31: 0xbead91d1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 29: 0xc4550116 (stale) Mirror 30: 0xe20eea22 (stale) Mirror 31: 0xbead91d1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000952 1788569991.791491 read 0x002def thru 0x00d2b0 (0xa4c2 bytes) 000953 1788569991.832409 write 0x004845 thru 0x007ea4 (0x3660 bytes) 000956 1788569991.840507 mapwrite 0x01e100 thru 0x01f2c1 (0x11c2 bytes) 000957 1788569991.900328 read 0x005102 thru 0x00d0e1 (0x7fe0 bytes) 000958 1788569991.902108 trunc from 0x040000 to 0x009fe8 (0x36017 bytes) 000959 1788569991.936131 trunc from 0x009fe9 to 0x0134c0 (0x94d8 bytes) 000960 1788569991.985430 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x134c1): Mirror 30: 0x6ec6c0ed (stale) Mirror 31: 0x24250363 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x134c1): Mirror 30: 0x6ec6c0ed (stale) Mirror 31: 0x24250363 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x134c1: skipped 000961 1788569992.375036 mapwrite 0x02f095 thru 0x034590 (0x54fc bytes) 000962 1788569992.549053 mapwrite 0x03e714 thru 0x03ffff (0x18ec bytes) 000963 1788569992.642989 trunc from 0x040000 to 0x038373 (0x7c8c bytes) 000965 1788569992.688808 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38374): Mirror 30: 0x73931795 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38374): Mirror 30: 0x73931795 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38374: skipped 000966 1788569993.071761 punch from 0x03307c to 0x038373 (0x52f8 bytes) 000967 1788569993.083616 trunc from 0x038374 to 0x03ef85 (0x6c12 bytes) 000969 1788569993.128024 punch from 0x02755c to 0x029dfa (0x289f bytes) 000970 1788569993.143073 trunc from 0x03ef86 to 0x00fc64 (0x2f321 bytes) 000971 1788569993.186766 punch from 0x008abf to 0x008af5 (0x37 bytes) 000972 1788569993.202623 read 0x00eabe thru 0x00fc64 (0x11a7 bytes) 000974 1788569993.254900 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xfc65): Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xfc65): Mirror 30: 0x5910248f (stale) CRC-32 checksum value for chunk [0, 0xfc65): Mirror 1: 0xdb9cbc39 Mirror 31: 0xdb9cbc39 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xfc65: skipped 000975 1788569994.393186 mapread 0x00a17d thru 0x00ab3c (0x9c0 bytes) 000978 1788569994.410286 write 0x0146ac thru 0x01a4d1 (0x5e26 bytes) 000979 1788569994.568072 mapwrite 0x01ae2e thru 0x0288b7 (0xda8a bytes) 000980 1788569994.697128 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x288b8): Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x288b8): Mirror 30: 0x7c3005f4 (stale) Mirror 31: 0x2000ce53 (stale) CRC-32 checksum value for chunk [0, 0x288b8): Mirror 1: 0x4bc9f493 Mirror 32: 0x4bc9f493 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x288b8: skipped 000982 1788569996.082748 mapwrite 0x0145bb thru 0x019ed1 (0x5917 bytes) 000984 1788569996.347108 read 0x01ff92 thru 0x0288b7 (0x8926 bytes) 000985 1788569996.367985 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x288b8): Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x288b8): Mirror 30: 0x7c3005f4 (stale) Mirror 31: 0x2000ce53 (stale) Mirror 32: 0x4bc9f493 (stale) CRC-32 checksum value for chunk [0, 0x288b8): Mirror 1: 0x9c4b8603 Mirror 33: 0x9c4b8603 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x288b8: skipped 000987 1788569997.874260 write 0x03e1e1 thru 0x03ffff (0x1e1f bytes) 000989 1788569997.960367 punch from 0x001346 to 0x0065c7 (0x5282 bytes) 000991 1788569997.973434 write 0x00a3a6 thru 0x01306a (0x8cc5 bytes) 000992 1788569997.985344 trunc from 0x040000 to 0x02eb1c (0x114e3 bytes) 000993 1788569998.018901 trunc from 0x02eb1d to 0x0127cb (0x1c351 bytes) 000994 1788569998.049934 write 0x01d887 thru 0x022b20 (0x529a bytes) 000995 1788569998.051922 write 0x034ce5 thru 0x03a6e8 (0x5a04 bytes) 000996 1788569998.066780 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3a6e9): Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a6e9): Mirror 30: 0xf6e341c8 (stale) Mirror 31: 0xc255444d (stale) Mirror 32: 0x3194238b (stale) Mirror 33: 0x82a15f7f (stale) CRC-32 checksum value for chunk [0, 0x3a6e9): Mirror 1: 0xb956b7f2 Mirror 34: 0xb956b7f2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a6e9: skipped 000997 1788569999.464651 mapwrite 0x0262b0 thru 0x02ff54 (0x9ca5 bytes) 000998 1788569999.674843 read 0x01704d thru 0x017a2a (0x9de bytes) 000999 1788569999.680790 read 0x00e6ae thru 0x0111fa (0x2b4d bytes) 001000 1788569999.693543 read 0x004247 thru 0x00d47e (0x9238 bytes) All operations completed A-OK! PASS 70a (184s) == sanity-flr test 71: check mirror extend parent fid ==== 21:00:05 (1788570005) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 21:00:09 (1788570009) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.184618 s, 11.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ad:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167925 s, 6.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ad:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ad:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ae:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 21:00:20 (1788570020) Starting client: oleg406-client.virtnet: -o user_xattr,flock 192.168.204.106@tcp:/lustre /mnt/lustre2 Starting client: oleg406-client.virtnet: -o user_xattr,flock 192.168.204.106@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 5319520 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3603776 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7353472 .. failed Extending file size to 7553152 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4255840 .. Extending file size to 4267072 .. Extending file size to 6197888 .. Extending file size to 6598656 .. Extending file size to 7506080 .. Extending file size to 9317632 .. Extending file size to 8621088 .. Extending file size to 8933600 .. Extending file size to 9223968 .. Waiting 57465 57466 57467 57468 57470 done 192.168.204.106@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg406-client.virtnet /mnt/lustre2 (opts:) 192.168.204.106@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg406-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (84s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 21:01:45 (1788570105) Starting client: oleg406-client.virtnet: -o user_xattr,flock 192.168.204.106@tcp:/lustre /mnt/lustre2 Starting client: oleg406-client.virtnet: -o user_xattr,flock 192.168.204.106@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1483072 .. Extending file size to 6236224 .. done Extending file size to 8328640 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7848672 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8489952 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8455360 .. Extending file size to 9222784 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 59098 59099 59100 59104 59105 192.168.204.106@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg406-client.virtnet /mnt/lustre2 (opts:) 192.168.204.106@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg406-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (84s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 21:03:10 (1788570190) Starting client: oleg406-client.virtnet: -o user_xattr,flock 192.168.204.106@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.157242 s, 13.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.348393 s, 6.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x5a2c056c (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.106@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg406-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 21:03:30 (1788570210) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.109861 s, 19.1 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 21:03:37 (1788570217) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ea:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2df:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2df:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e0:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2c3:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 21:03:45 (1788570225) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.206017 s, 10.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c5:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 21:03:53 (1788570233) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.295864 s, 7.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 21:04:03 (1788570243) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.224043 s, 9.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c9:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 21:04:38 (1788570278) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg406-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112966 s, 9.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e5:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ca:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e5:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ca:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e6:0x0] } PASS 204d (65s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 21:05:43 (1788570343) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg406-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.192098 s, 10.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (63s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 21:06:46 (1788570406) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg406-server mds-ost sync done. Creating new pool test_204f oleg406-server: Pool lustre.test_204f created Adding targets to pool oleg406-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg406-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.172931 s, 12.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg406-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg406-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg406-server: Pool lustre.test_204f destroyed PASS 204f (84s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 21:08:10 (1788570490) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 21:08:19 (1788570499) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 21:08:27 (1788570507) oleg406-server: Pool lustre.test_206 created Adding targets to pool oleg406-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg406-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg406-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg406-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg406-server: Pool lustre.test_206 destroyed PASS 206 (28s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 21:08:55 (1788570535) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103619 s, 10.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327121 s, 32.1 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x916f2bbc Mirror 3: 0x916f2bbc PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 21:09:08 (1788570548) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 21:09:13 (1788570553) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 21:09:16 (1788570556) Starting client: oleg406-client.virtnet: -o user_xattr,flock 192.168.204.106@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0924643 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xb50000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xa30000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xa30000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xa30000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xb50000 Read this (4 bytes): BBBB 192.168.204.106@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg406-client.virtnet /mnt/lustre2 (opts:) PASS 209a (44s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 21:10:00 (1788570600) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0354766 s, 231 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 21:10:08 (1788570608) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709874 s, 14.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x20f lmm_fid: [0x200000404:0x20f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x20f lmm_fid: [0x200000404:0x20f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d8:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834447 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 21:10:10.000000000 -0400 Modify: 2026-09-04 21:10:10.000000000 -0400 Change: 2026-09-04 21:10:11.000000000 -0400 Birth: 2026-09-04 21:10:10.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 21:10:18 (1788570618) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 21:10:22 (1788570622) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.664402 s, 61.6 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0587748 s, 69.7 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 4 21:10 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x211:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 4 21:10 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 21:10:32 (1788570632) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 21:10:40 (1788570640) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 2046 sec =========== 21:10:51 (1788570651) === sanity-flr: start cleanup 21:10:52 (1788570652) === === sanity-flr: finish cleanup 21:11:00 (1788570660) ===