-----============= acceptance-small: sanity-flr ============----- Mon Aug 10 18:38:03 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg235-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 18:38:23 (1786401503) === oleg235-client.virtnet: executing check_config_client /mnt/lustre oleg235-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg235-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff994f07dac000.idle_timeout=debug osc.lustre-OST0001-osc-ffff994f07dac000.idle_timeout=debug disable quota as required oleg235-server: oleg235-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 18:39:10 (1786401550) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8213] == sanity-flr test 0a: lfs mirror create with -N option == 18:39:12 (1786401552) PASS 0a (17s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 18:39:29 (1786401569) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 18:39:32 (1786401572) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 18:39:36 (1786401576) PASS 0d (22s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 18:39:58 (1786401598) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 18:40:02 (1786401602) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 18:40:05 (1786401605) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff994f07dac000.stats=clear osc.lustre-OST0001-osc-ffff994f07dac000.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 18:40:15 (1786401615) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 18:40:26 (1786401626) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 18:40:34 (1786401634) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 18:40:37 (1786401637) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 18:40:45 (1786401645) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 18:40:54 (1786401654) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 18:41:02 (1786401662) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 18:41:09 (1786401669) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 18:41:16 (1786401676) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 18:41:26 (1786401686) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 18:41:33 (1786401693) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 1.10173 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100673 s, 10.4 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 18:41:42 (1786401702) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.259137 s, 36.4 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 18:41:50 (1786401710) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 18:41:59 (1786401719) Stopping /mnt/lustre-ost1 (opts:) on oleg235-server oleg235-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff994f07dac000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff994f07dac000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg235-server: oleg235-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg235-server: oleg235-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg235-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg235-server oleg235-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff994f07dac000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff994f07dac000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg235-server: oleg235-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg235-server: oleg235-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg235-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (81s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 18:43:20 (1786401800) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.314333 s, 23.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg235-server oleg235-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff994f07dac000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff994f07dac000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg235-server: oleg235-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg235-server: oleg235-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg235-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (41s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 18:44:01 (1786401841) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg235-server oleg235-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff994f07dac000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff994f07dac000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg235-server: oleg235-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg235-server: oleg235-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg235-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg235-server oleg235-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff994f07dac000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff994f07dac000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg235-server: oleg235-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg235-server: oleg235-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg235-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (83s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 18:45:24 (1786401924) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.23662 s, 16.2 MB/s -rw-r--r-- 1 root root 52428800 Aug 10 18:45 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.962s user 0m0.001s sys 0m0.193s Stopping /mnt/lustre-ost1 (opts:) on oleg235-server oleg235-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff994f07dac000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff994f07dac000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.998s user 0m0.001s sys 0m0.170s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg235-server: oleg235-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg235-server: oleg235-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg235-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg235-server oleg235-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff994f07dac000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff994f07dac000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.974s user 0m0.000s sys 0m0.172s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg235-server: oleg235-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg235-server: oleg235-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg235-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg235-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff994f07dac000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff994f07dac000.ost_server_uuid in FULL state after 0 sec PASS 33b (96s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 18:47:00 (1786402020) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 18:47:04 (1786402024) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 18:47:07 (1786402027) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 18:47:11 (1786402031) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 18:47:19 (1786402039) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.504549 s, 24.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 18:47:31 (1786402051) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013415 s, 78.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214977 s, 48.8 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 18:47:39 (1786402059) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 18:47:48 (1786402068) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00179095 s, 572 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00129712 s, 789 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00158094 s, 648 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00155794 s, 657 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 18:47:58 (1786402078) create test files with size 15738813 .. 65dcbc4e15c1fbbd029242757a6423d1 - 6e16a37c8da98a92134a0397c47b8fb0 - 23fd19c36259a19bcadb5fe1248e1a02 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg235-client.virtnet /mnt/lustre (opts:) Starting client: oleg235-client.virtnet: -o user_xattr,flock 192.168.202.135@tcp:/lustre /mnt/lustre PASS 37 (25s) == sanity-flr test 38: resync ============================ 18:48:23 (1786402103) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 18:48:38 (1786402118) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 18:48:46 (1786402126) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574378 s, 18.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xae:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xaf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xcd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472244 s, 22.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 18:48:55 (1786402135) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0654751 s, 32.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.347905 s, 12.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.19624 s, 10.7 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 18:49:12 (1786402152) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 18:49:16 (1786402156) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 18:49:19 (1786402159) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405574 s, 25.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (10s) == sanity-flr test 44a: lfs mirror split check =========== 18:49:29 (1786402169) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.269912 s, 11.7 MB/s PASS 44a (10s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 18:49:39 (1786402179) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb9:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 18:49:49 (1786402189) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.181232 s, 57.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg235-client.virtnet /mnt/lustre (opts:) Starting client: oleg235-client.virtnet: -o user_xattr,flock 192.168.202.135@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 18:50:02 (1786402202) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff994f11b70800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.541005 s, 19.4 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg235-client.virtnet /mnt/lustre (opts:) Starting client: oleg235-client.virtnet: -o user_xattr,flock 192.168.202.135@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff994f07dac800.xattr_cache=1 PASS 44e (21s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 18:50:23 (1786402223) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (38s) == sanity-flr test 46: Verify setstripe --copy option ==== 18:51:01 (1786402261) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (35s) == sanity-flr test 47: Verify mirror obj alloc =========== 18:51:37 (1786402297) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 18:51:40 (1786402300) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.263705 s, 11.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.163235 s, 19.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x11bef8a7 (nosync,stale) Mirror 2: 0x11bef8a7 (nosync,stale) ** verify mirror 2 doesn't change original checksum: a1ba62de6836eccf4f0c708a5e958341 - mirror 1 checksum: 301b7484ba8b4c151475e25ab9a9cee5 - mirror 2 checksum: a1ba62de6836eccf4f0c708a5e958341 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-10T18:51:42-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xe3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-10T18:51:42-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xe4:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: a1ba62de6836eccf4f0c708a5e958341 - mirror 1 checksum: 301b7484ba8b4c151475e25ab9a9cee5 - mirror 2 checksum: 301b7484ba8b4c151475e25ab9a9cee5 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-10T18:51:42-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xe3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-10T18:51:42-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xe4:0x0] } PASS 48 (10s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 18:51:50 (1786402310) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342336 s, 30.6 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 18:51:58 (1786402318) Failing mds1 on oleg235-server Stopping /mnt/lustre-mds1 (opts:) on oleg235-server 18:52:10 (1786402330) shut down facet: mds1 facet_host: oleg235-server facet_failover_host: oleg235-server Failover mds1 to oleg235-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg235-server: oleg235-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 Started lustre-MDT0000 18:52:45 (1786402365) targets are mounted 18:52:45 (1786402365) facet_failover done oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822968 s, 12.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xe6:0x0] } PASS 50A (67s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 18:53:05 (1786402385) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0878123 s, 233 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0123734 s, 1.7 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (14s) == sanity-flr test 50b: mirror rsync handles sparseness == 18:53:19 (1786402399) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.106919 s, 192 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0151129 s, 1.4 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (13s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 18:53:32 (1786402412) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0111991 s, 1.5 MB/s PASS 50c (10s) == sanity-flr test 50d: mirror rsync keep holes ========== 18:53:42 (1786402422) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0341301 s, 600 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (10s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 18:53:52 (1786402432) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00792401 s, 517 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0351514 s, 117 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 18:54:00 (1786402440) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 18:54:23 (1786402463) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 18:54:51 (1786402491) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 18:55:15 (1786402515) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (28s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 18:55:43 (1786402543) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 18:55:52 (1786402552) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 57359 mirror split pid 57360 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 57359 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 57360 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 18:57:01 (1786402621) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 2624 skipping zero size read skipping zero length punch hole skipping zero size read 000007 1786402624.468765 mapwrite 0x0264f2 thru 0x035769 (0xf278 bytes) truncating to largest ever: 0x2f465 000008 1786402624.593150 trunc from 0x03576a to 0x02f464 (0x6305 bytes) 000009 1786402624.619399 read 0x01f753 thru 0x023651 (0x3eff bytes) 000010 1786402624.630809 mapwrite 0x01809e thru 0x020147 (0x80aa bytes) 000011 1786402624.667989 fallocate 0x011468 thru 0x0212d3 (0xfe6c bytes) 000012 1786402624.674226 mapread 0x004a14 thru 0x010469 (0xba56 bytes) 000013 1786402624.694088 mapwrite 0x003c11 thru 0x007e07 (0x41f7 bytes) 000014 1786402624.736306 trunc from 0x02f465 to 0x005727 (0x29d3d bytes) 000015 1786402624.762264 punch from 0x003666 to 0x005727 (0x20c2 bytes) 000016 1786402624.778703 mapwrite 0x0243e2 thru 0x02a71c (0x633b bytes) 000017 1786402624.847149 trunc from 0x02a71d to 0x014d05 (0x15a17 bytes) 000018 1786402624.927870 trunc from 0x004130 to 0x004439 (0x30a bytes) fallocating to largest ever: 0x3a531 000021 1786402624.942199 fallocate 0x035911 thru 0x03a530 (0x4c20 bytes) 000022 1786402624.956525 mapread 0x022b90 thru 0x031260 (0xe6d1 bytes) 000023 1786402625.026825 trunc from 0x01703f to 0x018f4d (0x1f0f bytes) 000025 1786402625.045473 write 0x01506d thru 0x01c414 (0x73a8 bytes) 000026 1786402625.056846 fallocate 0x014311 thru 0x022904 (0xe5f4 bytes) 000027 1786402625.073479 write 0x0351eb thru 0x03e4ec (0x9302 bytes) 000028 1786402625.090771 fallocate 0x002592 thru 0x00693d (0x43ac bytes) 000029 1786402625.106704 punch from 0x0257bb to 0x02aa85 (0x52cb bytes) 000030 1786402625.117297 trunc from 0x03e4ed to 0x016f09 (0x275e3 bytes) 000031 1786402625.156604 fallocate 0x014b72 thru 0x01b83d (0x6ccc bytes) 000033 1786402625.164737 write 0x01d82f thru 0x02a7de (0xcfb0 bytes) 000034 1786402625.168151 trunc from 0x015883 to 0x022c97 (0xd415 bytes) 000035 1786402625.185739 punch from 0x028f38 to 0x029f8b (0x1054 bytes) 000036 1786402625.197964 punch from 0x0288d0 to 0x02a7de (0x1f0f bytes) 000037 1786402625.219764 MIRROR_EXTEND to 2 mirrors 000038 1786402626.421103 trunc from 0x02a7df to 0x025640 (0x519e bytes) 000039 1786402626.500553 trunc from 0x023450 to 0x02400c (0xbbd bytes) 000040 1786402626.511567 read 0x010783 thru 0x013dd0 (0x364e bytes) 000041 1786402626.524480 trunc from 0x0236b3 to 0x02535f (0x1cad bytes) 000042 1786402626.536866 trunc from 0x025641 to 0x01edff (0x6841 bytes) 000043 1786402626.567641 fallocate 0x02f3cc thru 0x02f5ba (0x1ef bytes) 000044 1786402626.575713 trunc from 0x0290e3 to 0x02f5ba (0x64d8 bytes) 000045 1786402626.586255 punch from 0x003c1d to 0x007a44 (0x3e28 bytes) 000046 1786402626.599704 trunc from 0x007d23 to 0x01198b (0x9c69 bytes) 000047 1786402626.605924 trunc from 0x029ce8 to 0x02f5ba (0x58d3 bytes) 000048 1786402626.614717 punch from 0x026547 to 0x027d2d (0x17e7 bytes) 000049 1786402626.643707 punch from 0x01bec5 to 0x01db82 (0x1cbe bytes) 000050 1786402626.655526 punch from 0x01bb3f to 0x024cc3 (0x9185 bytes) 000051 1786402626.669529 fallocate 0x01b45b thru 0x023ca2 (0x8848 bytes) 000052 1786402626.676626 trunc from 0x02f5bb to 0x000fdd (0x2e5dd bytes) 000053 1786402626.713598 write 0x035daa thru 0x03ffff (0xa256 bytes) 000054 1786402626.723786 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x5dca5093 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9b4d7321 Mirror 3: 0x9b4d7321 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000055 1786402627.989262 fallocate 0x015d17 thru 0x01ce43 (0x712d bytes) 000056 1786402628.040269 read 0x02954d thru 0x036900 (0xd3b4 bytes) 000058 1786402628.050712 trunc from 0x01e7f0 to 0x02ad4e (0xc55f bytes) 000059 1786402628.056668 write 0x01cfd8 thru 0x024755 (0x777e bytes) 000060 1786402628.075960 mapread 0x02f559 thru 0x03b1b6 (0xbc5e bytes) 000061 1786402628.101859 trunc from 0x031594 to 0x0317db (0x248 bytes) fallocating to largest ever: 0x40000 000062 1786402628.108416 fallocate 0x03fcf3 thru 0x03ffff (0x30d bytes) 000063 1786402628.119496 MIRROR_RESYNC 3 mirrors 000064 1786402629.296295 trunc from 0x021099 to 0x024c43 (0x3bab bytes) 000065 1786402629.359496 trunc from 0x040000 to 0x02d530 (0x12acf bytes) 000066 1786402629.433420 mapwrite 0x003e6f thru 0x010f9d (0xd12f bytes) 000067 1786402629.506035 trunc from 0x027e30 to 0x02be56 (0x4027 bytes) 000068 1786402629.518923 punch from 0x00fb70 to 0x01fad4 (0xff65 bytes) 000069 1786402629.526875 read 0x01b351 thru 0x028151 (0xce01 bytes) 000070 1786402629.541963 write 0x00854c thru 0x00fb15 (0x75ca bytes) 000071 1786402629.568783 MIRROR_RESYNC 3 mirrors 000072 1786402631.173427 trunc from 0x02d531 to 0x03da99 (0x10569 bytes) 000073 1786402631.276493 write 0x01022e thru 0x016274 (0x6047 bytes) 000074 1786402631.296257 read 0x02a147 thru 0x034389 (0xa243 bytes) 000075 1786402631.318306 punch from 0x0192a0 to 0x02174c (0x84ad bytes) 000076 1786402631.324956 fallocate 0x00532c thru 0x015066 (0xfd3b bytes) 000077 1786402631.331537 fallocate 0x00d892 thru 0x014608 (0x6d77 bytes) 000078 1786402631.338579 mapwrite 0x02fd65 thru 0x03e902 (0xeb9e bytes) 000080 1786402631.411972 MIRROR_RESYNC 3 mirrors 000081 1786402632.705226 punch from 0x0002f8 to 0x009247 (0x8f50 bytes) 000082 1786402632.749279 read 0x009509 thru 0x01860b (0xf103 bytes) 000083 1786402632.760564 mapwrite 0x0102d7 thru 0x01ce19 (0xcb43 bytes) 000085 1786402632.804665 read 0x00df9b thru 0x0110a1 (0x3107 bytes) 000087 1786402632.811761 MIRROR_RESYNC 3 mirrors 000088 1786402634.044577 mapread 0x03c02d thru 0x03e902 (0x28d6 bytes) 000089 1786402634.129749 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18e:0x0]: Invalid argument (22) 000090 1786402634.848692 trunc from 0x03e903 to 0x0266c6 (0x1823c bytes) 000091 1786402634.973550 trunc from 0x0266c7 to 0x03d859 (0x17193 bytes) 000092 1786402635.012632 punch from 0x03b5cf to 0x03d859 (0x228b bytes) 000093 1786402635.025543 punch from 0x015a36 to 0x024823 (0xedee bytes) 000094 1786402635.034828 trunc from 0x03d85a to 0x00d754 (0x30105 bytes) 000095 1786402635.087075 read 0x00a9ec thru 0x00c0fb (0x1710 bytes) 000096 1786402635.097593 punch from 0x000b20 to 0x00d754 (0xcc35 bytes) 000097 1786402635.104338 read 0x00c85e thru 0x00d754 (0xef7 bytes) 000098 1786402635.165088 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xd755): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd755): Mirror 3: 0x850baa22 (stale) CRC-32 checksum value for chunk [0, 0xd755): Mirror 1: 0x740a5b8d Mirror 4: 0x740a5b8d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd755: skipped 000099 1786402636.197473 fallocate 0x021570 thru 0x02d929 (0xc3ba bytes) 000100 1786402636.273852 fallocate 0x034a08 thru 0x03639f (0x1998 bytes) 000101 1786402636.289881 punch from 0x00623a to 0x0110ad (0xae74 bytes) 000102 1786402636.304558 fallocate 0x024b40 thru 0x024eba (0x37b bytes) 000103 1786402636.321936 trunc from 0x02d92a to 0x025196 (0x8793 bytes) 000104 1786402636.415401 fallocate 0x01fa02 thru 0x0290ae (0x96ad bytes) 000105 1786402636.427098 mapread 0x01c6d6 thru 0x02347b (0x6da6 bytes) 000106 1786402636.510829 read 0x0073b8 thru 0x00b544 (0x418d bytes) 000107 1786402636.529643 trunc from 0x0012ac to 0x006e12 (0x5b67 bytes) 000108 1786402636.542882 mapwrite 0x0218ad thru 0x0233c0 (0x1b14 bytes) 000109 1786402636.571149 write 0x03e1cf thru 0x03ffff (0x1e31 bytes) 000110 1786402636.575700 trunc from 0x01fa96 to 0x02cfcf (0xd53a bytes) 000111 1786402636.587092 read 0x03acce thru 0x03ffff (0x5332 bytes) 000112 1786402636.618632 mapwrite 0x033b73 thru 0x03c531 (0x89bf bytes) 000113 1786402636.681629 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x5527f595 (stale) Mirror 4: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7d6e9f27 Mirror 5: 0x7d6e9f27 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000114 1786402637.950752 mapread 0x01052c thru 0x0204dd (0xffb2 bytes) 000115 1786402637.987420 fallocate 0x031a52 thru 0x03ffff (0xe5ae bytes) 000116 1786402638.038556 mapwrite 0x016c9f thru 0x023664 (0xc9c6 bytes) 000117 1786402638.089330 fallocate 0x0003cf thru 0x00f303 (0xef35 bytes) 000118 1786402638.095473 trunc from 0x0079dd to 0x015e7d (0xe4a1 bytes) 000119 1786402638.102465 fallocate 0x02fd9d thru 0x0335c3 (0x3827 bytes) 000121 1786402638.111821 read 0x032027 thru 0x03ffff (0xdfd9 bytes) 000122 1786402638.120861 read 0x0290f7 thru 0x02a597 (0x14a1 bytes) 000123 1786402638.132035 trunc from 0x026d07 to 0x02db92 (0x6e8c bytes) 000124 1786402638.152819 fallocate 0x01ca7c thru 0x01d695 (0xc1a bytes) 000125 1786402638.163948 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x5527f595 (stale) Mirror 4: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x5527f595 (stale) Mirror 4: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000126 1786402638.653511 punch from 0x037078 to 0x03c17b (0x5104 bytes) 000127 1786402638.660717 mapwrite 0x004d98 thru 0x01064f (0xb8b8 bytes) 000128 1786402638.700644 punch from 0x02b53b to 0x02bc2b (0x6f1 bytes) 000129 1786402638.706655 punch from 0x00d567 to 0x01149d (0x3f37 bytes) 000130 1786402638.713332 trunc from 0x00c001 to 0x014b97 (0x8b97 bytes) 000131 1786402638.719642 punch from 0x03cd65 to 0x03dba5 (0xe41 bytes) 000132 1786402638.725613 write 0x03548e thru 0x039271 (0x3de4 bytes) 000133 1786402638.742772 fallocate 0x0071cd thru 0x013c6c (0xcaa0 bytes) 000134 1786402638.749018 fallocate 0x00b7e8 thru 0x013879 (0x8092 bytes) 000136 1786402638.758868 MIRROR_RESYNC 3 mirrors 000137 1786402640.540122 trunc from 0x040000 to 0x034ada (0xb525 bytes) 000140 1786402640.605664 write 0x020c75 thru 0x024e88 (0x4214 bytes) 000141 1786402640.617155 fallocate 0x0063cf thru 0x0133e1 (0xd013 bytes) 000142 1786402640.630474 fallocate 0x018c11 thru 0x021fb2 (0x93a2 bytes) 000143 1786402640.638684 mapread 0x002584 thru 0x010bc9 (0xe646 bytes) 000144 1786402640.682877 trunc from 0x034adb to 0x030009 (0x4ad1 bytes) 000145 1786402640.720156 write 0x024974 thru 0x02dafe (0x918b bytes) 000146 1786402640.736957 MIRROR_RESYNC 3 mirrors 000147 1786402642.157751 write 0x0281fa thru 0x036a60 (0xe867 bytes) 000148 1786402642.216139 trunc from 0x026416 to 0x027f96 (0x1b81 bytes) 000149 1786402642.231941 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36a61): Mirror 4: 0x15762d4e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36a61): Mirror 4: 0x15762d4e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36a61: skipped 000150 1786402642.620629 read 0x00a033 thru 0x013402 (0x93d0 bytes) 000151 1786402642.641958 punch from 0x032ed2 to 0x036a60 (0x3b8f bytes) 000152 1786402642.651067 trunc from 0x02c714 to 0x02f50b (0x2df8 bytes) 000153 1786402642.655668 read 0x003d37 thru 0x0118db (0xdba5 bytes) 000154 1786402642.679832 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x36a61): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36a61): Mirror 4: 0x15762d4e (stale) CRC-32 checksum value for chunk [0, 0x36a61): Mirror 1: 0xd84ece10 Mirror 5: 0xd84ece10 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36a61: skipped 000155 1786402643.909529 mapwrite 0x02e16e thru 0x03be7d (0xdd10 bytes) 000156 1786402644.109254 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3be7e): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3be7e): Mirror 4: 0x93b6d0bd (stale) Mirror 5: 0x471382e8 (stale) CRC-32 checksum value for chunk [0, 0x3be7e): Mirror 1: 0xa8617df Mirror 6: 0xa8617df lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3be7e: skipped 000157 1786402645.505255 fallocate 0x01dbb0 thru 0x02b4e8 (0xd939 bytes) 000158 1786402645.560193 MIRROR_RESYNC 4 mirrors 000160 1786402647.098136 punch from 0x010fc3 to 0x01896b (0x79a9 bytes) 000161 1786402647.162522 punch from 0x032a6f to 0x035337 (0x28c9 bytes) 000162 1786402647.175414 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3be7e): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3be7e): Mirror 4: 0xa8617df (stale) Mirror 5: 0xa8617df (stale) Mirror 6: 0xa8617df (stale) CRC-32 checksum value for chunk [0, 0x3be7e): Mirror 1: 0x450d1b1d Mirror 7: 0x450d1b1d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3be7e: skipped 000163 1786402648.650729 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3be7e): Mirror 4: 0xa8617df (stale) Mirror 6: 0xa8617df (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3be7e): Mirror 4: 0xa8617df (stale) Mirror 6: 0xa8617df (stale) CRC-32 checksum value for chunk [0, 0x3be7e): Mirror 1: 0x450d1b1d Mirror 7: 0x450d1b1d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3be7e: skipped 000164 1786402649.179684 punch from 0x036c9b to 0x038982 (0x1ce8 bytes) 000165 1786402649.243863 mapwrite 0x036813 thru 0x03ffff (0x97ed bytes) 000167 1786402649.328689 trunc from 0x040000 to 0x0066dd (0x39922 bytes) 000168 1786402649.344848 punch from 0x002bd3 to 0x0066dd (0x3b0b bytes) 000169 1786402649.351533 mapwrite 0x008e11 thru 0x011a15 (0x8c05 bytes) 000170 1786402649.412983 fallocate 0x02dccc thru 0x03a099 (0xc3ce bytes) 000172 1786402649.421216 fallocate 0x03d416 thru 0x03db1c (0x707 bytes) 000173 1786402649.426699 trunc from 0x011243 to 0x0200fe (0xeebc bytes) 000174 1786402649.435056 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3db1d): Mirror 4: 0x134ff766 (stale) Mirror 6: 0x134ff766 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3db1d): Mirror 4: 0x134ff766 (stale) Mirror 6: 0x134ff766 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3db1d: skipped 000175 1786402649.936817 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3db1d): Mirror 6: 0x134ff766 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3db1d): Mirror 6: 0x134ff766 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3db1d: skipped 000176 1786402650.456817 fallocate 0x022d47 thru 0x02c033 (0x92ed bytes) 000177 1786402650.466170 trunc from 0x03db1d to 0x01741e (0x266fe bytes) 000178 1786402650.484803 trunc from 0x01741f to 0x039474 (0x22056 bytes) 000179 1786402650.507214 punch from 0x0382ef to 0x039474 (0x1186 bytes) 000180 1786402650.518364 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18e:0x0]: Invalid argument (22) 000181 1786402650.712297 trunc from 0x039475 to 0x01afd5 (0x1e49f bytes) 000182 1786402650.744760 trunc from 0x01afd6 to 0x0133b0 (0x7c25 bytes) 000183 1786402650.777616 write 0x0131d8 thru 0x019f4c (0x6d75 bytes) 000184 1786402650.785281 mapread 0x00e504 thru 0x019399 (0xae96 bytes) 000185 1786402650.810679 punch from 0x015f78 to 0x019d8b (0x3e14 bytes) 000186 1786402650.832438 mapread 0x014292 thru 0x019f4c (0x5cbb bytes) 000187 1786402650.855369 trunc from 0x019f4d to 0x00056c (0x199e0 bytes) 000188 1786402650.882038 trunc from 0x00013d to 0x00056c (0x430 bytes) 000189 1786402650.891792 trunc from 0x00054e to 0x00056c (0x1f bytes) 000191 1786402650.897771 trunc from 0x0003b3 to 0x00056c (0x1ba bytes) 000192 1786402650.908045 mapread 0x0004ad thru 0x00056c (0xc0 bytes) 000195 1786402650.923206 punch from 0x000312 to 0x00056c (0x25b bytes) 000196 1786402650.932007 punch from 0x000408 to 0x00056c (0x165 bytes) 000197 1786402650.943662 mapread 0x000553 thru 0x00056c (0x1a bytes) 000198 1786402650.956551 read 0x00016f thru 0x00056c (0x3fe bytes) 000201 1786402650.959542 trunc from 0x00056d to 0x007b09 (0x759d bytes) 000202 1786402650.982190 trunc from 0x007b0a to 0x03c23d (0x34734 bytes) 000204 1786402650.998679 trunc from 0x026770 to 0x027ed1 (0x1762 bytes) 000205 1786402651.005871 mapwrite 0x0328cf thru 0x0349f4 (0x2126 bytes) 000206 1786402651.028700 mapread 0x001efd thru 0x00b759 (0x985d bytes) 000207 1786402651.046831 mapwrite 0x02c201 thru 0x032415 (0x6215 bytes) 000208 1786402651.072691 write 0x008bc7 thru 0x0142e7 (0xb721 bytes) 000209 1786402651.079782 trunc from 0x00c37d to 0x01a8c8 (0xe54c bytes) 000210 1786402651.094963 fallocate 0x02a05c thru 0x02b48c (0x1431 bytes) 000212 1786402651.106845 MIRROR_EXTEND to 2 mirrors 000213 1786402651.711510 read 0x00f727 thru 0x01f466 (0xfd40 bytes) 000214 1786402651.723467 trunc from 0x0340c1 to 0x03c23d (0x817d bytes) 000215 1786402651.775020 fallocate 0x02459b thru 0x033727 (0xf18d bytes) 000216 1786402651.783703 read 0x003871 thru 0x0049c4 (0x1154 bytes) 000217 1786402651.796714 trunc from 0x03c23e to 0x00dbb8 (0x2e685 bytes) 000219 1786402651.845988 trunc from 0x000972 to 0x004b29 (0x41b8 bytes) 000220 1786402651.852361 read 0x0052b5 thru 0x00ad22 (0x5a6e bytes) 000221 1786402651.853344 punch from 0x004f5f to 0x00dbb8 (0x8c5a bytes) 000222 1786402651.860043 trunc from 0x00dbb9 to 0x00b2e3 (0x28d5 bytes) 000223 1786402651.878676 fallocate 0x030e70 thru 0x03ffff (0xf190 bytes) 000224 1786402651.884825 trunc from 0x00b5f7 to 0x00e50d (0x2f17 bytes) 000225 1786402651.890435 mapread 0x015ca0 thru 0x01fa15 (0x9d76 bytes) 000226 1786402651.931384 read 0x02c82b thru 0x0310d3 (0x48a9 bytes) 000227 1786402651.932178 fallocate 0x012b06 thru 0x01ba86 (0x8f81 bytes) 000228 1786402651.943857 MIRROR_RESYNC 2 mirrors 000230 1786402652.819173 trunc from 0x040000 to 0x007501 (0x38afe bytes) 000231 1786402652.938824 MIRROR_RESYNC 2 mirrors 000232 1786402653.729258 write 0x03e5ff thru 0x03f4d0 (0xed2 bytes) 000233 1786402653.820025 trunc from 0x03f4d1 to 0x022017 (0x1d4b9 bytes) 000234 1786402653.893562 mapwrite 0x02b4e3 thru 0x0391f5 (0xdd13 bytes) 000235 1786402653.981871 punch from 0x014ccd to 0x019e42 (0x5176 bytes) 000236 1786402653.994991 punch from 0x00bc14 to 0x017167 (0xb554 bytes) 000237 1786402654.006830 read 0x02c25e thru 0x035684 (0x9427 bytes) 000238 1786402654.011760 trunc from 0x0368d9 to 0x0391f5 (0x291d bytes) 000239 1786402654.017871 fallocate 0x000381 thru 0x00d255 (0xced5 bytes) 000240 1786402654.028552 read 0x01eae6 thru 0x025d88 (0x72a3 bytes) 000241 1786402654.054475 fallocate 0x00c1b2 thru 0x01bfe7 (0xfe36 bytes) 000242 1786402654.065189 write 0x02183e thru 0x026e7c (0x563f bytes) 000243 1786402654.069492 punch from 0x022f9c to 0x023253 (0x2b8 bytes) 000244 1786402654.088256 mapwrite 0x02d672 thru 0x035e9d (0x882c bytes) 000245 1786402654.118285 trunc from 0x0391f6 to 0x02b7bf (0xda36 bytes) 000246 1786402654.163155 MIRROR_RESYNC 2 mirrors 000247 1786402654.734390 punch from 0x018774 to 0x01b0f0 (0x297d bytes) 000248 1786402654.774652 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18e:0x0]: Invalid argument (22) 000249 1786402654.964532 MIRROR_EXTEND to 2 mirrors 000251 1786402656.174033 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18e:0x0]: Invalid argument (22) 000252 1786402656.359967 mapwrite 0x02b5d8 thru 0x036699 (0xb0c2 bytes) 000253 1786402656.563714 trunc from 0x015a1c to 0x01cb05 (0x70ea bytes) 000254 1786402656.579555 read 0x028cd4 thru 0x02e944 (0x5c71 bytes) 000255 1786402656.581139 punch from 0x02bc1d to 0x02dd2a (0x210e bytes) 000256 1786402656.593980 fallocate 0x0341d8 thru 0x03cd98 (0x8bc1 bytes) 000257 1786402656.609116 mapread 0x01c2e6 thru 0x024082 (0x7d9d bytes) 000258 1786402656.647545 punch from 0x00f873 to 0x01ca2d (0xd1bb bytes) 000259 1786402656.655657 read 0x02e644 thru 0x034e27 (0x67e4 bytes) 000260 1786402656.656463 mapwrite 0x01e627 thru 0x025ed3 (0x78ad bytes) 000261 1786402656.776927 trunc from 0x01c2ee to 0x0231f4 (0x6f07 bytes) 000262 1786402656.795004 fallocate 0x0270b1 thru 0x02cda8 (0x5cf8 bytes) 000263 1786402656.808499 punch from 0x0186f5 to 0x022334 (0x9c40 bytes) 000264 1786402656.818103 mapwrite 0x01de68 thru 0x024a6a (0x6c03 bytes) 000265 1786402656.903625 trunc from 0x03cd99 to 0x008307 (0x34a91 bytes) 000266 1786402656.931533 fallocate 0x026e7b thru 0x030ffb (0xa181 bytes) 000267 1786402656.946431 write 0x021e12 thru 0x02cc86 (0xae75 bytes) 000268 1786402656.950355 punch from 0x00c871 to 0x013f96 (0x7726 bytes) 000269 1786402656.957985 read 0x00cf3a thru 0x0123eb (0x54b2 bytes) 000270 1786402656.994574 MIRROR_EXTEND to 2 mirrors 000271 1786402658.015600 fallocate 0x0018f3 thru 0x00bdb1 (0xa4bf bytes) 000272 1786402658.136974 write 0x003832 thru 0x00948c (0x5c5b bytes) 000273 1786402658.175965 mapread 0x004979 thru 0x007cdc (0x3364 bytes) 000274 1786402658.185224 mapread 0x00a112 thru 0x00f702 (0x55f1 bytes) 000275 1786402658.238120 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18e:0x0]: Invalid argument (22) 000277 1786402658.459870 fallocate 0x01a233 thru 0x022104 (0x7ed2 bytes) 000278 1786402658.466359 read 0x00da1b thru 0x01d730 (0xfd16 bytes) 000279 1786402658.477477 trunc from 0x02cc87 to 0x006fc8 (0x25cbe bytes) 000280 1786402658.497845 read 0x00474c thru 0x0061e8 (0x1a9d bytes) 000281 1786402658.509525 write 0x03c2f7 thru 0x03ffff (0x3d09 bytes) 000282 1786402658.513541 fallocate 0x01dbcc thru 0x025340 (0x7775 bytes) 000283 1786402658.519444 mapwrite 0x03f87a thru 0x03ffff (0x786 bytes) 000284 1786402658.533548 write 0x02cc48 thru 0x03ad67 (0xe120 bytes) 000285 1786402658.545764 write 0x000c2d thru 0x00436e (0x3742 bytes) 000286 1786402658.547011 read 0x016213 thru 0x01cc31 (0x6a1f bytes) 000287 1786402658.548460 mapread 0x00358e thru 0x0093ed (0x5e60 bytes) 000288 1786402658.550244 read 0x039d1d thru 0x03ffff (0x62e3 bytes) 000289 1786402658.551092 fallocate 0x00479a thru 0x00afeb (0x6852 bytes) 000290 1786402658.560500 MIRROR_EXTEND to 2 mirrors 000291 1786402659.475629 write 0x01663f thru 0x01a521 (0x3ee3 bytes) 000292 1786402659.525590 punch from 0x01099f to 0x01b201 (0xa863 bytes) 000293 1786402659.541837 fallocate 0x000bd9 thru 0x00b100 (0xa528 bytes) 000294 1786402659.547780 trunc from 0x040000 to 0x017dfb (0x28204 bytes) 000295 1786402659.571840 fallocate 0x0268fc thru 0x027680 (0xd85 bytes) 000296 1786402659.582524 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18e:0x0]: Invalid argument (22) 000297 1786402659.805858 punch from 0x00bfaf to 0x0155c9 (0x961b bytes) 000299 1786402659.825684 trunc from 0x017dfc to 0x0354bf (0x1d6c4 bytes) 000301 1786402659.854413 trunc from 0x00bf39 to 0x00eeee (0x2fb6 bytes) 000302 1786402659.872946 fallocate 0x02a6c9 thru 0x0343da (0x9d12 bytes) 000303 1786402659.882989 fallocate 0x0209a7 thru 0x02180e (0xe68 bytes) 000304 1786402659.905254 fallocate 0x029890 thru 0x038ead (0xf61e bytes) 000305 1786402659.912698 mapread 0x00237b thru 0x00ffa4 (0xdc2a bytes) 000306 1786402659.968606 mapread 0x033d5c thru 0x0354bf (0x1764 bytes) 000307 1786402659.996087 mapwrite 0x025df3 thru 0x035894 (0xfaa2 bytes) 000309 1786402660.090481 mapwrite 0x00e3d1 thru 0x01abe4 (0xc814 bytes) 000310 1786402660.147378 mapwrite 0x004896 thru 0x00887c (0x3fe7 bytes) 000311 1786402660.171831 mapread 0x001fd4 thru 0x00bbed (0x9c1a bytes) 000313 1786402660.175634 mapwrite 0x00e3b1 thru 0x01d762 (0xf3b2 bytes) 000314 1786402660.207275 mapwrite 0x03576c thru 0x03e4b6 (0x8d4b bytes) 000315 1786402660.265615 write 0x006cb4 thru 0x010ba9 (0x9ef6 bytes) 000316 1786402660.268796 punch from 0x03961c to 0x03e4b6 (0x4e9b bytes) 000317 1786402660.276957 mapwrite 0x02f56b thru 0x03d64b (0xe0e1 bytes) 000318 1786402660.315360 mapread 0x0279d4 thru 0x0323f6 (0xaa23 bytes) 000320 1786402660.318538 mapread 0x00c0ba thru 0x019b7d (0xdac4 bytes) 000322 1786402660.322977 punch from 0x015408 to 0x01dd6f (0x8968 bytes) 000324 1786402660.332651 mapread 0x029921 thru 0x02de1e (0x44fe bytes) 000325 1786402660.334328 write 0x022750 thru 0x032408 (0xfcb9 bytes) 000326 1786402660.337177 trunc from 0x03e4b7 to 0x024c1b (0x1989b bytes) 000327 1786402660.361250 trunc from 0x024c1c to 0x015e57 (0xedc4 bytes) 000328 1786402660.381592 write 0x01df38 thru 0x01fb71 (0x1c3a bytes) 000329 1786402660.384565 mapread 0x016b6e thru 0x01740d (0x8a0 bytes) 000330 1786402660.395955 mapread 0x00d8e6 thru 0x013f17 (0x6632 bytes) 000331 1786402660.398498 trunc from 0x001c08 to 0x002bcf (0xfc8 bytes) 000333 1786402660.404830 trunc from 0x01fb72 to 0x00816c (0x17a05 bytes) 000334 1786402660.422482 mapwrite 0x02a1e5 thru 0x038f8f (0xedab bytes) 000335 1786402660.480417 mapread 0x03677c thru 0x038f8f (0x2814 bytes) 000336 1786402660.482376 fallocate 0x036b52 thru 0x039ab1 (0x2f60 bytes) 000337 1786402660.488362 fallocate 0x0302a5 thru 0x03dd6d (0xdac9 bytes) 000338 1786402660.493747 trunc from 0x038f90 to 0x01ca00 (0x1c58f bytes) 000340 1786402660.512390 trunc from 0x01ca01 to 0x010cf9 (0xbd07 bytes) 000341 1786402660.531832 punch from 0x00b879 to 0x00c5a1 (0xd29 bytes) 000343 1786402660.541207 mapread 0x00d053 thru 0x0101da (0x3188 bytes) 000344 1786402660.566547 read 0x000aa8 thru 0x003e0b (0x3364 bytes) 000345 1786402660.575710 trunc from 0x010cfa to 0x03453e (0x23845 bytes) 000346 1786402660.597422 write 0x0271b2 thru 0x02b210 (0x405f bytes) 000347 1786402660.610601 read 0x00a2fd thru 0x00edae (0x4ab2 bytes) 000348 1786402660.611772 read 0x0068ff thru 0x015bb0 (0xf2b2 bytes) 000349 1786402660.625971 mapwrite 0x034993 thru 0x03e165 (0x97d3 bytes) 000350 1786402660.684781 mapwrite 0x0164eb thru 0x01b6c6 (0x51dc bytes) 000351 1786402660.704162 trunc from 0x011f30 to 0x01e23e (0xc30f bytes) 000352 1786402660.708415 trunc from 0x035a32 to 0x037a2f (0x1ffe bytes) 000353 1786402660.714225 trunc from 0x03e166 to 0x01dd70 (0x203f5 bytes) 000354 1786402660.729498 fallocate 0x001e7f thru 0x004d29 (0x2eab bytes) 000356 1786402660.735789 fallocate 0x022210 thru 0x02e668 (0xc459 bytes) 000357 1786402660.741253 trunc from 0x018540 to 0x01b8fc (0x33bd bytes) 000358 1786402660.763648 mapwrite 0x03eb14 thru 0x03ffff (0x14ec bytes) 000359 1786402660.819938 trunc from 0x01ee09 to 0x0213cd (0x25c5 bytes) 000361 1786402660.826845 fallocate 0x0294cf thru 0x0356cb (0xc1fd bytes) 000362 1786402660.833393 read 0x02a3a2 thru 0x038859 (0xe4b8 bytes) 000363 1786402660.847874 mapwrite 0x02be5c thru 0x03403a (0x81df bytes) 000364 1786402660.873188 write 0x0194b3 thru 0x028868 (0xf3b6 bytes) 000365 1786402660.875960 mapread 0x03fed1 thru 0x03ffff (0x12f bytes) 000368 1786402660.881494 mapread 0x02b2ef thru 0x039187 (0xde99 bytes) 000369 1786402660.883136 fallocate 0x0039d3 thru 0x00545f (0x1a8d bytes) 000370 1786402660.890466 trunc from 0x03708c to 0x03ffff (0x8f74 bytes) 000371 1786402660.896863 trunc from 0x024c4a to 0x030f50 (0xc307 bytes) 000372 1786402660.914827 write 0x037abd thru 0x03ffff (0x8543 bytes) 000373 1786402660.923242 write 0x038e00 thru 0x039d0e (0xf0f bytes) 000374 1786402660.925764 punch from 0x03851b to 0x03f2c0 (0x6da6 bytes) 000375 1786402660.941297 fallocate 0x00f8a1 thru 0x01d094 (0xd7f4 bytes) 000376 1786402660.946943 read 0x02b1ee thru 0x02b538 (0x34b bytes) 000378 1786402660.957681 punch from 0x00d053 to 0x012443 (0x53f1 bytes) 000380 1786402660.964597 read 0x010eb6 thru 0x012ef7 (0x2042 bytes) 000381 1786402660.976270 fallocate 0x007c1c thru 0x0101a6 (0x858b bytes) 000382 1786402660.982276 fallocate 0x01c212 thru 0x024168 (0x7f57 bytes) 000383 1786402660.988343 punch from 0x00a596 to 0x00da84 (0x34ef bytes) 000384 1786402660.993659 fallocate 0x02966e thru 0x038486 (0xee19 bytes) 000385 1786402660.999622 fallocate 0x0215b2 thru 0x02c6af (0xb0fe bytes) 000386 1786402661.006953 punch from 0x009ab5 to 0x00f829 (0x5d75 bytes) 000387 1786402661.014633 punch from 0x0396c9 to 0x03ffff (0x6937 bytes) 000388 1786402661.020453 fallocate 0x019426 thru 0x0231e3 (0x9dbe bytes) 000389 1786402661.026527 mapwrite 0x038803 thru 0x03ffff (0x77fd bytes) 000391 1786402661.100806 trunc from 0x0219f1 to 0x02e75c (0xcd6c bytes) 000392 1786402661.107822 read 0x039870 thru 0x03ffff (0x6790 bytes) 000393 1786402661.109810 trunc from 0x024167 to 0x0307f8 (0xc692 bytes) 000394 1786402661.116855 mapread 0x0064fe thru 0x0145bb (0xe0be bytes) 000396 1786402661.132797 read 0x02adce thru 0x02f93e (0x4b71 bytes) 000397 1786402661.133706 read 0x03e764 thru 0x03ffff (0x189c bytes) 000399 1786402661.137692 write 0x0177f1 thru 0x020ea4 (0x96b4 bytes) 000400 1786402661.146677 MIRROR_EXTEND to 2 mirrors 000401 1786402662.004605 read 0x03ac6e thru 0x03ffff (0x5392 bytes) 000402 1786402662.027487 fallocate 0x0144b2 thru 0x01e808 (0xa357 bytes) 000404 1786402662.114918 mapread 0x016909 thru 0x023917 (0xd00f bytes) 000405 1786402662.173419 trunc from 0x040000 to 0x02b161 (0x14e9e bytes) 000406 1786402662.252587 write 0x03fb14 thru 0x03ffff (0x4ec bytes) 000407 1786402662.254014 read 0x008d4a thru 0x00b9c7 (0x2c7e bytes) 000408 1786402662.307473 read 0x03e5d3 thru 0x03ffff (0x1a2d bytes) 000409 1786402662.308623 punch from 0x01a309 to 0x027cd2 (0xd9ca bytes) 000410 1786402662.324083 read 0x00822b thru 0x00d8b8 (0x568e bytes) 000411 1786402662.325170 fallocate 0x03ce2c thru 0x03e5e0 (0x17b5 bytes) 000412 1786402662.331824 mapread 0x018a47 thru 0x01fab5 (0x706f bytes) 000413 1786402662.357266 trunc from 0x03f427 to 0x03ffff (0xbd9 bytes) 000414 1786402662.390968 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc76dc201 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x70c5d14 Mirror 3: 0x70c5d14 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000415 1786402663.703448 trunc from 0x029b14 to 0x034a95 (0xaf82 bytes) 000416 1786402663.736210 punch from 0x0079c1 to 0x009145 (0x1785 bytes) 000417 1786402663.741174 mapwrite 0x03742c thru 0x03ffff (0x8bd4 bytes) 000418 1786402663.785578 read 0x027fa1 thru 0x0365d6 (0xe636 bytes) 000420 1786402663.803118 mapwrite 0x018ba5 thru 0x01b691 (0x2aed bytes) 000421 1786402663.845925 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x70c5d14 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x70c5d14 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000422 1786402664.163115 punch from 0x018612 to 0x022671 (0xa060 bytes) 000425 1786402664.173112 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x70c5d14 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x90433c0c Mirror 4: 0x90433c0c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000426 1786402665.320657 read 0x008431 thru 0x015531 (0xd101 bytes) 000427 1786402665.347695 fallocate 0x0281f5 thru 0x0308f8 (0x8704 bytes) 000428 1786402665.425383 punch from 0x012f88 to 0x01c2e9 (0x9362 bytes) 000430 1786402665.439058 trunc from 0x040000 to 0x02c340 (0x13cbf bytes) 000431 1786402665.490659 trunc from 0x017c37 to 0x01e26e (0x6638 bytes) 000432 1786402665.495788 mapwrite 0x001995 thru 0x00ea21 (0xd08d bytes) 000433 1786402665.591359 punch from 0x01deb7 to 0x029102 (0xb24c bytes) 000434 1786402665.604577 MIRROR_RESYNC 3 mirrors 000436 1786402666.737743 mapwrite 0x00c519 thru 0x010aac (0x4594 bytes) 000437 1786402666.850673 write 0x03d130 thru 0x03ffff (0x2ed0 bytes) 000438 1786402666.852694 trunc from 0x040000 to 0x0259f7 (0x1a608 bytes) 000440 1786402666.883679 punch from 0x00ccdc to 0x01abe5 (0xdf0a bytes) 000442 1786402666.890800 write 0x02ff2b thru 0x0351bf (0x5295 bytes) 000443 1786402666.892868 read 0x026530 thru 0x02bd7a (0x584b bytes) 000444 1786402666.907755 read 0x0281a6 thru 0x02f57d (0x73d8 bytes) 000445 1786402666.927776 fallocate 0x03867f thru 0x03ffff (0x7981 bytes) 000447 1786402666.954737 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x351c0): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x351c0): Mirror 3: 0x80d44881 (stale) Mirror 4: 0x80d44881 (stale) CRC-32 checksum value for chunk [0, 0x351c0): Mirror 1: 0x1fe5ed70 Mirror 5: 0x1fe5ed70 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x351c0: skipped 000448 1786402667.904756 read 0x01a6d0 thru 0x02339b (0x8ccc bytes) 000449 1786402667.922017 write 0x030232 thru 0x03fa3e (0xf80d bytes) 000450 1786402667.969470 read 0x028872 thru 0x02ae5a (0x25e9 bytes) 000452 1786402667.982700 mapread 0x01d67a thru 0x01e9da (0x1361 bytes) 000453 1786402667.984050 trunc from 0x03fa3f to 0x02a284 (0x157ba bytes) 000455 1786402668.004565 read 0x01407e thru 0x020702 (0xc685 bytes) 000456 1786402668.006637 punch from 0x016417 to 0x02120a (0xadf4 bytes) 000457 1786402668.013123 mapread 0x0155f5 thru 0x01b8aa (0x62b6 bytes) 000458 1786402668.023712 read 0x006609 thru 0x00fd62 (0x975a bytes) 000459 1786402668.024734 fallocate 0x01aec6 thru 0x01b336 (0x471 bytes) 000460 1786402668.030264 trunc from 0x02a285 to 0x00994b (0x20939 bytes) 000461 1786402668.045460 fallocate 0x02fddd thru 0x03e863 (0xea87 bytes) 000462 1786402668.060486 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e864): Mirror 3: 0x1d10ff5a (stale) Mirror 5: 0x9a26f1fd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e864): Mirror 3: 0x1d10ff5a (stale) Mirror 5: 0x9a26f1fd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e864: skipped 000463 1786402668.466450 write 0x034960 thru 0x039b48 (0x51e9 bytes) 000464 1786402668.486677 fallocate 0x03c741 thru 0x03e34e (0x1c0e bytes) 000465 1786402668.492440 fallocate 0x00564a thru 0x009c6c (0x4623 bytes) 000466 1786402668.497729 write 0x00db39 thru 0x00dc07 (0xcf bytes) 000467 1786402668.510678 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e864): Mirror 3: 0x1d10ff5a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e864): Mirror 3: 0x1d10ff5a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e864: skipped 000468 1786402668.860301 punch from 0x001d57 to 0x005683 (0x392d bytes) 000469 1786402668.866717 mapwrite 0x01ef64 thru 0x026311 (0x73ae bytes) 000470 1786402668.902469 fallocate 0x03a90c thru 0x03f11b (0x4810 bytes) 000471 1786402668.908379 write 0x00c5b6 thru 0x0102f8 (0x3d43 bytes) 000472 1786402668.926709 trunc from 0x03f11c to 0x03f3ef (0x2d4 bytes) 000473 1786402668.955356 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3f3f0): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f3f0): Mirror 3: 0x2b006936 (stale) CRC-32 checksum value for chunk [0, 0x3f3f0): Mirror 1: 0x4933f57a Mirror 4: 0x4933f57a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f3f0: skipped 000474 1786402670.169915 fallocate 0x03e5d8 thru 0x03ffff (0x1a28 bytes) 000475 1786402670.257832 trunc from 0x03f3f0 to 0x00c0b7 (0x33338 bytes) 000476 1786402670.326709 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xc0b8): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc0b8): Mirror 3: 0x2e2d8719 (stale) Mirror 4: 0x57266657 (stale) CRC-32 checksum value for chunk [0, 0xc0b8): Mirror 1: 0x57266657 Mirror 5: 0x57266657 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc0b8: skipped 000477 1786402671.648140 write 0x000aa3 thru 0x0033ed (0x294b bytes) 000478 1786402671.708236 read 0x004950 thru 0x00c0b7 (0x7768 bytes) 000480 1786402671.719336 mapwrite 0x02aab5 thru 0x034a08 (0x9f54 bytes) 000481 1786402671.779754 mapwrite 0x03d399 thru 0x03ffff (0x2c67 bytes) 000482 1786402671.816627 mapwrite 0x03f131 thru 0x03ffff (0xecf bytes) 000483 1786402671.829218 mapread 0x02be80 thru 0x02e544 (0x26c5 bytes) 000484 1786402671.830544 read 0x0316b5 thru 0x03d0ac (0xb9f8 bytes) 000486 1786402671.852996 mapread 0x006c8b thru 0x00e695 (0x7a0b bytes) 000487 1786402671.854834 punch from 0x00133f to 0x00faa7 (0xe769 bytes) 000488 1786402671.862250 mapwrite 0x029ef2 thru 0x02af46 (0x1055 bytes) 000489 1786402671.875695 punch from 0x010ca3 to 0x013050 (0x23ae bytes) 000490 1786402671.881367 punch from 0x0200a7 to 0x0298d6 (0x9830 bytes) 000491 1786402671.895795 MIRROR_RESYNC 4 mirrors 000492 1786402672.975837 punch from 0x00528f to 0x00a9a2 (0x5714 bytes) 000493 1786402673.012394 trunc from 0x00322a to 0x00d16c (0x9f43 bytes) 000494 1786402673.018565 read 0x028d46 thru 0x02b79b (0x2a56 bytes) 000495 1786402673.025285 mapread 0x00ac02 thru 0x00b163 (0x562 bytes) 000497 1786402673.038119 mapread 0x018bfb thru 0x021dd0 (0x91d6 bytes) 000499 1786402673.064730 fallocate 0x0320fd thru 0x039a23 (0x7927 bytes) 000500 1786402673.069823 read 0x0016c9 thru 0x00cf31 (0xb869 bytes) 000502 1786402673.081201 trunc from 0x03bd38 to 0x03ffff (0x42c8 bytes) 000503 1786402673.086585 write 0x03f505 thru 0x03ffff (0xafb bytes) 000504 1786402673.096238 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xb0e2bb14 (stale) Mirror 5: 0xb0e2bb14 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xb0e2bb14 (stale) Mirror 5: 0xb0e2bb14 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000505 1786402673.427304 read 0x03cccc thru 0x03ffff (0x3334 bytes) 000506 1786402673.443714 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xb0e2bb14 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xb0e2bb14 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000507 1786402673.789787 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xb0e2bb14 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x290567f9 Mirror 6: 0x290567f9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000508 1786402674.655620 write 0x003c5f thru 0x010cb9 (0xd05b bytes) 000509 1786402674.709079 trunc from 0x040000 to 0x034c51 (0xb3ae bytes) 000510 1786402674.743291 punch from 0x00f7b2 to 0x01cd12 (0xd561 bytes) 000511 1786402674.763677 fallocate 0x022ce9 thru 0x02c63b (0x9953 bytes) 000512 1786402674.771298 mapwrite 0x02b0e7 thru 0x038a25 (0xd93f bytes) 000513 1786402674.862099 trunc from 0x0237fa to 0x029142 (0x5949 bytes) 000515 1786402674.869171 fallocate 0x02d5c4 thru 0x03c555 (0xef92 bytes) 000516 1786402674.874600 mapwrite 0x004762 thru 0x00acfa (0x6599 bytes) 000517 1786402674.892505 write 0x025279 thru 0x02ab92 (0x591a bytes) 000518 1786402674.898948 mapwrite 0x01a679 thru 0x021e3d (0x77c5 bytes) 000519 1786402674.931702 read 0x02ad52 thru 0x039dbd (0xf06c bytes) 000520 1786402674.938336 mapwrite 0x016453 thru 0x023ce8 (0xd896 bytes) 000521 1786402674.968222 trunc from 0x006952 to 0x008e2b (0x24da bytes) 000523 1786402674.981518 MIRROR_RESYNC 3 mirrors 000524 1786402676.133694 mapwrite 0x029065 thru 0x02c938 (0x38d4 bytes) 000525 1786402676.278631 mapwrite 0x004153 thru 0x007fb4 (0x3e62 bytes) 000526 1786402676.355532 mapwrite 0x026f9c thru 0x02d0a0 (0x6105 bytes) 000528 1786402676.410656 read 0x028a82 thru 0x031ef0 (0x946f bytes) 000529 1786402676.417565 mapwrite 0x032922 thru 0x034189 (0x1868 bytes) 000530 1786402676.490234 trunc from 0x03c556 to 0x0160ab (0x264aa bytes) 000531 1786402676.511550 fallocate 0x014fca thru 0x01d783 (0x87ba bytes) 000532 1786402676.522672 read 0x00059a thru 0x00dc92 (0xd6f9 bytes) 000533 1786402676.546430 read 0x01d70c thru 0x01d783 (0x78 bytes) 000535 1786402676.548157 fallocate 0x033989 thru 0x03ffff (0xc677 bytes) 000536 1786402676.560467 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d784): Mirror 5: 0xa1008dd7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d784): Mirror 5: 0xa1008dd7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d784: skipped 000537 1786402676.825148 write 0x02ee99 thru 0x02f4a1 (0x609 bytes) 000538 1786402676.829200 mapwrite 0x02d33a thru 0x034584 (0x724b bytes) 000539 1786402676.880330 mapwrite 0x01686b thru 0x024516 (0xdcac bytes) 000542 1786402676.936033 trunc from 0x032527 to 0x034584 (0x205e bytes) 000543 1786402676.942348 trunc from 0x00acea to 0x00decf (0x31e6 bytes) 000544 1786402676.948420 read 0x00874b thru 0x00c2bf (0x3b75 bytes) 000545 1786402676.958626 read 0x022260 thru 0x0237a3 (0x1544 bytes) 000546 1786402676.960350 trunc from 0x0294b7 to 0x02a5eb (0x1135 bytes) 000547 1786402676.965345 mapwrite 0x001f3b thru 0x00e9bd (0xca83 bytes) 000548 1786402677.000381 trunc from 0x0129f9 to 0x01ad18 (0x8320 bytes) 000549 1786402677.006390 mapwrite 0x02d43e thru 0x03c8d6 (0xf499 bytes) 000550 1786402677.056765 fallocate 0x01063e thru 0x010d4f (0x712 bytes) 000551 1786402677.063260 mapwrite 0x02f5fb thru 0x03054c (0xf52 bytes) 000552 1786402677.077525 trunc from 0x03c8d7 to 0x0066e2 (0x361f4 bytes) 000554 1786402677.097437 trunc from 0x002f87 to 0x006580 (0x35fa bytes) 000555 1786402677.102793 read 0x0058c5 thru 0x005c12 (0x34e bytes) 000556 1786402677.112760 fallocate 0x01203e thru 0x015894 (0x3857 bytes) 000557 1786402677.118649 punch from 0x001e56 to 0x00d14d (0xb2f8 bytes) 000559 1786402677.125492 trunc from 0x0051a6 to 0x00ea09 (0x9864 bytes) 000560 1786402677.130448 write 0x03c00c thru 0x03ffff (0x3ff4 bytes) 000561 1786402677.132787 trunc from 0x022c7f to 0x0313da (0xe75c bytes) 000562 1786402677.137925 read 0x02c682 thru 0x02ea69 (0x23e8 bytes) 000563 1786402677.155415 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18e:0x0]: Invalid argument (22) 000564 1786402677.341338 punch from 0x01a302 to 0x029242 (0xef41 bytes) 000565 1786402677.349341 fallocate 0x0046d4 thru 0x006849 (0x2176 bytes) 000566 1786402677.355274 punch from 0x02da87 to 0x02e253 (0x7cd bytes) 000568 1786402677.366342 mapread 0x00ed44 thru 0x01db89 (0xee46 bytes) 000569 1786402677.392570 mapread 0x0232c1 thru 0x031198 (0xded8 bytes) 000570 1786402677.414116 punch from 0x030dfa to 0x032316 (0x151d bytes) 000571 1786402677.419701 trunc from 0x0095d8 to 0x00df8a (0x49b3 bytes) 000572 1786402677.425859 mapwrite 0x01d68c thru 0x025c9e (0x8613 bytes) 000573 1786402677.443822 trunc from 0x040000 to 0x02c0cc (0x13f33 bytes) 000574 1786402677.463608 mapwrite 0x01edea thru 0x029abf (0xacd6 bytes) 000575 1786402677.486699 trunc from 0x02c0cd to 0x0024f7 (0x29bd5 bytes) 000576 1786402677.506697 mapread 0x000ec8 thru 0x0024f7 (0x1630 bytes) 000577 1786402677.513239 fallocate 0x03e6a5 thru 0x03ffff (0x195b bytes) 000578 1786402677.518790 read 0x03e1d8 thru 0x03ffff (0x1e28 bytes) 000579 1786402677.525349 fallocate 0x010875 thru 0x01dde6 (0xd572 bytes) 000580 1786402677.530652 fallocate 0x01745d thru 0x01a117 (0x2cbb bytes) 000581 1786402677.536763 write 0x00c336 thru 0x0122c0 (0x5f8b bytes) 000582 1786402677.548310 read 0x011ba2 thru 0x01f31f (0xd77e bytes) 000583 1786402677.561350 trunc from 0x040000 to 0x03c351 (0x3cae bytes) 000584 1786402677.575274 mapread 0x01531e thru 0x015ea2 (0xb85 bytes) 000585 1786402677.576645 mapread 0x032e52 thru 0x03c351 (0x9500 bytes) 000586 1786402677.579483 fallocate 0x018914 thru 0x01a254 (0x1941 bytes) 000587 1786402677.585269 write 0x03e328 thru 0x03ffff (0x1cd8 bytes) 000588 1786402677.587207 trunc from 0x040000 to 0x03d483 (0x2b7c bytes) 000590 1786402677.607196 punch from 0x022457 to 0x031f5c (0xfb06 bytes) 000592 1786402677.614368 write 0x015af0 thru 0x0254af (0xf9c0 bytes) 000593 1786402677.621626 mapwrite 0x011235 thru 0x020077 (0xee43 bytes) 000594 1786402677.642732 write 0x010541 thru 0x01e6e4 (0xe1a4 bytes) 000595 1786402677.646478 trunc from 0x001c6f to 0x00601d (0x43af bytes) 000596 1786402677.653280 mapwrite 0x00dedf thru 0x01bb70 (0xdc92 bytes) 000597 1786402677.679641 trunc from 0x03d484 to 0x01544c (0x28037 bytes) 000598 1786402677.709534 mapwrite 0x032197 thru 0x03cd82 (0xabec bytes) 000599 1786402677.827555 mapread 0x02425c thru 0x032efa (0xec9f bytes) 000600 1786402677.898573 trunc from 0x02ba53 to 0x03a070 (0xe61e bytes) 000601 1786402677.905582 write 0x0353c3 thru 0x03ffff (0xac3d bytes) 000602 1786402677.922769 punch from 0x011a46 to 0x01dfda (0xc595 bytes) 000605 1786402677.949565 punch from 0x028d22 to 0x032982 (0x9c61 bytes) 000606 1786402677.958341 trunc from 0x007af3 to 0x00984c (0x1d5a bytes) 000607 1786402677.964666 mapread 0x03d8b8 thru 0x03ffff (0x2748 bytes) 000609 1786402677.968419 punch from 0x00bff7 to 0x00f2b7 (0x32c1 bytes) 000611 1786402677.979439 mapwrite 0x00097a thru 0x005126 (0x47ad bytes) 000612 1786402678.035707 write 0x02bb8d thru 0x03a3d1 (0xe845 bytes) 000613 1786402678.049267 fallocate 0x02eec4 thru 0x02f098 (0x1d5 bytes) 000615 1786402678.062719 punch from 0x008d5c to 0x0105cc (0x7871 bytes) 000616 1786402678.073624 write 0x02ca19 thru 0x039d6f (0xd357 bytes) 000617 1786402678.082432 mapwrite 0x03c93e thru 0x03ffff (0x36c2 bytes) 000619 1786402678.118363 write 0x015d13 thru 0x024c6f (0xef5d bytes) 000620 1786402678.132102 punch from 0x0333bf to 0x03a4bf (0x7101 bytes) 000621 1786402678.140974 write 0x026e2f thru 0x02779d (0x96f bytes) 000622 1786402678.142304 trunc from 0x00962c to 0x018af4 (0xf4c9 bytes) 000624 1786402678.165201 mapwrite 0x003d8f thru 0x01320f (0xf481 bytes) 000625 1786402678.347754 read 0x023acf thru 0x02c49d (0x89cf bytes) 000626 1786402678.388397 read 0x01e84e thru 0x0278e3 (0x9096 bytes) 000627 1786402678.389182 read 0x011efa thru 0x012a8a (0xb91 bytes) 000628 1786402678.389799 mapread 0x01dcdc thru 0x02663c (0x8961 bytes) 000629 1786402678.392552 write 0x034443 thru 0x03ffff (0xbbbd bytes) 000630 1786402678.395497 punch from 0x01c5cc to 0x020474 (0x3ea9 bytes) 000631 1786402678.406311 write 0x03d024 thru 0x03ffff (0x2fdc bytes) 000632 1786402678.411418 trunc from 0x040000 to 0x021688 (0x1e977 bytes) 000633 1786402678.453066 write 0x039a99 thru 0x03f183 (0x56eb bytes) 000634 1786402678.454652 read 0x03caf4 thru 0x03f183 (0x2690 bytes) 000635 1786402678.458720 trunc from 0x03197a to 0x03f183 (0xd80a bytes) 000637 1786402678.493211 fallocate 0x00dbe4 thru 0x016a30 (0x8e4d bytes) 000638 1786402678.508238 read 0x0200c2 thru 0x027a77 (0x79b6 bytes) 000639 1786402678.536464 trunc from 0x00bab8 to 0x011056 (0x559f bytes) 000640 1786402678.542319 write 0x0254b5 thru 0x02d26a (0x7db6 bytes) 000643 1786402678.548712 mapread 0x00f4c3 thru 0x015fc6 (0x6b04 bytes) 000644 1786402678.565410 read 0x01ce0c thru 0x02199e (0x4b93 bytes) 000645 1786402678.566334 punch from 0x02a478 to 0x039109 (0xec92 bytes) 000646 1786402678.580053 trunc from 0x03f184 to 0x03c876 (0x290d bytes) 000647 1786402678.596178 trunc from 0x02b232 to 0x03422d (0x8ffc bytes) 000648 1786402678.600279 trunc from 0x004b44 to 0x01197a (0xce37 bytes) 000649 1786402678.606053 trunc from 0x01e0e1 to 0x01f001 (0xf21 bytes) 000650 1786402678.610640 fallocate 0x034ecb thru 0x034f3c (0x72 bytes) 000653 1786402678.617065 mapread 0x00e3d5 thru 0x01bb99 (0xd7c5 bytes) 000654 1786402678.627115 fallocate 0x03db53 thru 0x03ffff (0x24ad bytes) 000655 1786402678.632024 punch from 0x012cc5 to 0x01e2e6 (0xb622 bytes) 000657 1786402678.638214 trunc from 0x027f6d to 0x037526 (0xf5ba bytes) 000658 1786402678.644522 mapread 0x005f61 thru 0x012d83 (0xce23 bytes) 000659 1786402678.656157 punch from 0x03592c to 0x038f94 (0x3669 bytes) 000660 1786402678.662794 mapwrite 0x01bb2b thru 0x028175 (0xc64b bytes) 000661 1786402678.780677 MIRROR_EXTEND to 2 mirrors 000662 1786402679.670119 read 0x02f876 thru 0x033ab3 (0x423e bytes) 000664 1786402679.682682 fallocate 0x0094ef thru 0x014b5c (0xb66e bytes) 000665 1786402679.722406 read 0x03bd05 thru 0x03ffff (0x42fb bytes) 000666 1786402679.737593 write 0x02e290 thru 0x03cdd5 (0xeb46 bytes) 000668 1786402679.746168 trunc from 0x01ac00 to 0x021f00 (0x7301 bytes) 000669 1786402679.752462 read 0x035d64 thru 0x03ffff (0xa29c bytes) 000670 1786402679.753465 write 0x02f3e5 thru 0x039fa8 (0xabc4 bytes) 000671 1786402679.758709 fallocate 0x018235 thru 0x01fe8b (0x7c57 bytes) 000672 1786402679.764314 write 0x0003e4 thru 0x004c03 (0x4820 bytes) 000673 1786402679.766447 trunc from 0x022fa9 to 0x02690a (0x3962 bytes) 000674 1786402679.773235 trunc from 0x03d564 to 0x03ffff (0x2a9c bytes) 000675 1786402679.783983 MIRROR_RESYNC 2 mirrors 000676 1786402680.601776 write 0x018fa0 thru 0x0228c9 (0x992a bytes) 000677 1786402680.648247 mapread 0x02363b thru 0x0288cd (0x5293 bytes) 000678 1786402680.673184 trunc from 0x040000 to 0x0009ab (0x3f654 bytes) 000679 1786402680.697386 mapread 0x0000e6 thru 0x0009ab (0x8c6 bytes) 000680 1786402680.705717 trunc from 0x0007b4 to 0x0009ab (0x1f8 bytes) 000681 1786402680.711377 punch from 0x000362 to 0x0009ab (0x64a bytes) 000682 1786402680.717131 mapread 0x000314 thru 0x0009ab (0x698 bytes) 000683 1786402680.727965 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18e:0x0]: Invalid argument (22) 000684 1786402680.910097 fallocate 0x009c54 thru 0x00f95c (0x5d09 bytes) 000685 1786402680.916154 mapwrite 0x02271f thru 0x025051 (0x2933 bytes) 000687 1786402680.978495 trunc from 0x0162e5 to 0x01ad21 (0x4a3d bytes) 000688 1786402680.985079 write 0x00c46a thru 0x01b70e (0xf2a5 bytes) 000689 1786402680.999634 read 0x00f5da thru 0x016f03 (0x792a bytes) 000690 1786402681.000687 trunc from 0x025052 to 0x033863 (0xe812 bytes) 000691 1786402681.021845 trunc from 0x00836f to 0x014dd1 (0xca63 bytes) 000692 1786402681.041078 mapread 0x01c992 thru 0x01d230 (0x89f bytes) 000693 1786402681.051444 write 0x00af32 thru 0x00fbe1 (0x4cb0 bytes) 000694 1786402681.063535 write 0x025620 thru 0x02eb6d (0x954e bytes) 000695 1786402681.071131 trunc from 0x033864 to 0x0065f8 (0x2d26b bytes) 000696 1786402681.095189 write 0x03f4b1 thru 0x03ffff (0xb4f bytes) 000697 1786402681.099024 mapread 0x00302f thru 0x00d973 (0xa945 bytes) 000698 1786402681.113155 mapwrite 0x001d18 thru 0x00febb (0xe1a4 bytes) 000699 1786402681.140875 mapwrite 0x03a0a8 thru 0x03ffff (0x5f58 bytes) 000701 1786402681.172763 mapread 0x013289 thru 0x014aaf (0x1827 bytes) 000702 1786402681.186183 punch from 0x022432 to 0x022f7e (0xb4d bytes) 000704 1786402681.191185 mapread 0x00816b thru 0x00d3df (0x5275 bytes) 000705 1786402681.192238 mapwrite 0x0336c0 thru 0x03ba83 (0x83c4 bytes) 000707 1786402681.211877 trunc from 0x040000 to 0x0322fa (0xdd05 bytes) 000709 1786402681.231379 write 0x010d5f thru 0x01a70b (0x99ad bytes) 000710 1786402681.234182 read 0x015618 thru 0x01d581 (0x7f6a bytes) 000711 1786402681.246715 trunc from 0x0090f6 to 0x017813 (0xe71e bytes) 000712 1786402681.261763 punch from 0x011791 to 0x020ecb (0xf73b bytes) 000713 1786402681.267486 trunc from 0x015c11 to 0x024fe4 (0xf3d4 bytes) 000714 1786402681.272363 fallocate 0x00244c thru 0x00c3b5 (0x9f6a bytes) 000715 1786402681.277318 mapwrite 0x00a414 thru 0x0161f7 (0xbde4 bytes) 000716 1786402681.358080 mapwrite 0x016454 thru 0x01c99a (0x6547 bytes) 000717 1786402681.404120 mapread 0x013cfd thru 0x0189bb (0x4cbf bytes) 000718 1786402681.406103 mapread 0x022bdd thru 0x02bc12 (0x9036 bytes) 000719 1786402681.426115 fallocate 0x0365f3 thru 0x03b44e (0x4e5c bytes) 000720 1786402681.438852 read 0x039e92 thru 0x03b44e (0x15bd bytes) 000721 1786402681.455283 punch from 0x02441b to 0x025e96 (0x1a7c bytes) 000723 1786402681.461202 mapread 0x021571 thru 0x023df5 (0x2885 bytes) 000725 1786402681.462754 punch from 0x02ba7f to 0x02fd82 (0x4304 bytes) 000726 1786402681.472024 punch from 0x013806 to 0x01fac3 (0xc2be bytes) 000728 1786402681.479075 read 0x0306f1 thru 0x0366b5 (0x5fc5 bytes) 000729 1786402681.479747 mapread 0x002dff thru 0x00bb42 (0x8d44 bytes) 000732 1786402681.485733 read 0x00e344 thru 0x010d38 (0x29f5 bytes) 000733 1786402681.486700 write 0x00ccb1 thru 0x012b2f (0x5e7f bytes) 000734 1786402681.488668 trunc from 0x03b44f to 0x02596f (0x15adf bytes) 000735 1786402681.518300 mapread 0x01ac8b thru 0x02596f (0xace5 bytes) 000737 1786402681.545862 MIRROR_EXTEND to 2 mirrors 000738 1786402682.324236 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18e:0x0]: Invalid argument (22) 000741 1786402682.629649 mapread 0x01b8c8 thru 0x02596f (0xa0a8 bytes) 000742 1786402682.689005 mapwrite 0x023ffe thru 0x0290a2 (0x50a5 bytes) 000743 1786402682.835271 mapread 0x017c25 thru 0x01d5f4 (0x59d0 bytes) 000744 1786402682.851192 trunc from 0x0290a3 to 0x00e7d5 (0x1a8cd bytes) 000745 1786402682.882218 mapread 0x00d610 thru 0x00e7d5 (0x11c6 bytes) 000746 1786402682.936203 fallocate 0x03cb85 thru 0x03ffff (0x347b bytes) 000749 1786402682.966509 trunc from 0x040000 to 0x00463a (0x3b9c5 bytes) 000750 1786402683.006174 trunc from 0x0041e7 to 0x00463a (0x454 bytes) 000751 1786402683.016142 trunc from 0x0000ef to 0x00463a (0x454c bytes) 000753 1786402683.028626 trunc from 0x00463b to 0x03c23d (0x37c03 bytes) 000754 1786402683.075676 punch from 0x025092 to 0x033056 (0xdfc5 bytes) 000755 1786402683.084285 mapwrite 0x021d84 thru 0x02d14d (0xb3ca bytes) 000756 1786402683.150418 MIRROR_EXTEND to 2 mirrors 000757 1786402684.322175 MIRROR_RESYNC 2 mirrors 000758 1786402684.851920 fallocate 0x01e578 thru 0x021367 (0x2df0 bytes) 000759 1786402684.888342 trunc from 0x03c23e to 0x018de4 (0x23459 bytes) 000760 1786402684.941560 write 0x00fcc4 thru 0x012aef (0x2e2c bytes) 000761 1786402684.951664 punch from 0x005aeb to 0x0138f3 (0xde09 bytes) 000762 1786402684.964253 read 0x004d40 thru 0x005632 (0x8f3 bytes) 000763 1786402684.971823 fallocate 0x03c6f9 thru 0x03ffff (0x3907 bytes) 000764 1786402684.979080 fallocate 0x018780 thru 0x01f026 (0x68a7 bytes) 000765 1786402684.983114 write 0x02f602 thru 0x030df7 (0x17f6 bytes) 000766 1786402684.985173 read 0x0149a0 thru 0x02242c (0xda8d bytes) 000767 1786402684.992676 mapread 0x011761 thru 0x0184e6 (0x6d86 bytes) 000768 1786402684.993592 trunc from 0x030df8 to 0x002c06 (0x2e1f1 bytes) 000769 1786402685.008608 mapwrite 0x008e73 thru 0x01477a (0xb908 bytes) 000770 1786402685.103867 fallocate 0x038862 thru 0x03ffff (0x779e bytes) 000771 1786402685.110552 trunc from 0x040000 to 0x013ecc (0x2c133 bytes) 000772 1786402685.129048 write 0x0245ff thru 0x02a5af (0x5fb1 bytes) 000773 1786402685.131655 mapread 0x01e618 thru 0x023ac0 (0x54a9 bytes) 000774 1786402685.169034 punch from 0x019d41 to 0x01f871 (0x5b31 bytes) 000775 1786402685.180690 fallocate 0x027db1 thru 0x0362f5 (0xe545 bytes) 000776 1786402685.186221 trunc from 0x02a5b0 to 0x03b27a (0x10ccb bytes) 000777 1786402685.218484 trunc from 0x01cca9 to 0x0232f0 (0x6648 bytes) 000778 1786402685.226471 write 0x03e6dd thru 0x03ffff (0x1923 bytes) 000779 1786402685.233859 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3067001d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xeea496e Mirror 3: 0xeea496e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000782 1786402686.468537 trunc from 0x00a5ff to 0x00bb3b (0x153d bytes) 000783 1786402686.520686 read 0x03e7be thru 0x03ffff (0x1842 bytes) 000784 1786402686.529102 trunc from 0x0265f7 to 0x029d1e (0x3728 bytes) 000785 1786402686.534370 mapwrite 0x0311b4 thru 0x033b32 (0x297f bytes) 000787 1786402686.574588 trunc from 0x040000 to 0x03df49 (0x20b6 bytes) 000788 1786402686.602020 mapwrite 0x027dc5 thru 0x02ea94 (0x6cd0 bytes) 000789 1786402686.632059 mapwrite 0x027d0d thru 0x032c47 (0xaf3b bytes) 000790 1786402686.651702 trunc from 0x0182b8 to 0x01fa6a (0x77b3 bytes) 000791 1786402686.660632 MIRROR_RESYNC 3 mirrors 000792 1786402687.562503 mapwrite 0x0193fd thru 0x0285d1 (0xf1d5 bytes) 000793 1786402687.726581 trunc from 0x01b76e to 0x026fc0 (0xb853 bytes) 000794 1786402687.735180 trunc from 0x02798c to 0x03695b (0xefd0 bytes) 000795 1786402687.743165 trunc from 0x01b6c0 to 0x01c709 (0x104a bytes) 000796 1786402687.750494 read 0x03984d thru 0x03bb17 (0x22cb bytes) 000797 1786402687.758606 punch from 0x014784 to 0x01a273 (0x5af0 bytes) 000798 1786402687.767225 read 0x01766b thru 0x02745f (0xfdf5 bytes) 000799 1786402687.781469 trunc from 0x03cb2a to 0x03df49 (0x1420 bytes) 000800 1786402687.793648 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3df4a): Mirror 3: 0x4655fc5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3df4a): Mirror 3: 0x4655fc5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3df4a: skipped 000801 1786402688.222524 fallocate 0x037db1 thru 0x03b822 (0x3a72 bytes) 000802 1786402688.242208 mapread 0x01f20f thru 0x024192 (0x4f84 bytes) 000803 1786402688.276206 mapwrite 0x02147a thru 0x02ed53 (0xd8da bytes) 000804 1786402688.342995 trunc from 0x03df4a to 0x015473 (0x28ad6 bytes) 000805 1786402688.371513 read 0x010c4d thru 0x015473 (0x4827 bytes) 000806 1786402688.418686 read 0x00fb33 thru 0x015473 (0x5941 bytes) 000807 1786402688.444067 trunc from 0x00cdcc to 0x015473 (0x86a8 bytes) 000808 1786402688.464641 mapread 0x0028e0 thru 0x0051fc (0x291d bytes) 000809 1786402688.466053 punch from 0x00c8da to 0x013e21 (0x7548 bytes) 000810 1786402688.482645 punch from 0x000283 to 0x00af31 (0xacaf bytes) 000811 1786402688.506582 MIRROR_RESYNC 2 mirrors 000812 1786402689.192317 mapread 0x000d73 thru 0x00c2d1 (0xb55f bytes) 000813 1786402689.217212 fallocate 0x01a13c thru 0x01ea14 (0x48d9 bytes) 000814 1786402689.256189 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18e:0x0]: Invalid argument (22) 000815 1786402689.395191 MIRROR_EXTEND to 2 mirrors 000816 1786402690.245566 mapwrite 0x01aa6b thru 0x0256ba (0xac50 bytes) 000817 1786402690.390236 trunc from 0x0256bb to 0x012ef0 (0x127ca bytes) 000820 1786402690.418230 mapwrite 0x03c2d1 thru 0x03ffff (0x3d2f bytes) 000821 1786402690.491819 MIRROR_RESYNC 2 mirrors 000822 1786402691.088006 trunc from 0x0020dc to 0x011a9e (0xf9c3 bytes) 000823 1786402691.117645 trunc from 0x040000 to 0x015692 (0x2a96d bytes) 000824 1786402691.152425 read 0x00fe30 thru 0x013893 (0x3a64 bytes) 000826 1786402691.158834 mapread 0x0140a2 thru 0x015692 (0x15f1 bytes) 000830 1786402691.170867 read 0x003f5b thru 0x011e09 (0xdeaf bytes) 000831 1786402691.180114 mapread 0x000d28 thru 0x002818 (0x1af1 bytes) 000833 1786402691.185670 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x15693): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15693): Mirror 2: 0x10154f75 (stale) CRC-32 checksum value for chunk [0, 0x15693): Mirror 1: 0x73bcede6 Mirror 3: 0x73bcede6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15693: skipped 000834 1786402691.942557 trunc from 0x002ff0 to 0x010d00 (0xdd11 bytes) 000835 1786402691.986148 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x15693): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15693): Mirror 2: 0x10154f75 (stale) Mirror 3: 0x73bcede6 (stale) CRC-32 checksum value for chunk [0, 0x15693): Mirror 1: 0x73bcede6 Mirror 4: 0x73bcede6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15693: skipped 000836 1786402692.970623 read 0x012dc1 thru 0x015692 (0x28d2 bytes) 000837 1786402692.977289 trunc from 0x015693 to 0x022a16 (0xd384 bytes) 000838 1786402693.033527 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22a17): Mirror 2: 0x90a0e41c (stale) Mirror 4: 0x6cbdb54e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22a17): Mirror 2: 0x90a0e41c (stale) Mirror 4: 0x6cbdb54e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22a17: skipped 000839 1786402693.309148 fallocate 0x0158d4 thru 0x022c87 (0xd3b4 bytes) 000842 1786402693.315733 punch from 0x013c55 to 0x01d50e (0x98ba bytes) 000843 1786402693.321354 fallocate 0x007e0b thru 0x01752a (0xf720 bytes) 000844 1786402693.327333 mapread 0x00aa7e thru 0x017176 (0xc6f9 bytes) 000845 1786402693.361142 punch from 0x00a1b4 to 0x0177a1 (0xd5ee bytes) 000846 1786402693.365805 mapwrite 0x038e1f thru 0x03c225 (0x3407 bytes) 000847 1786402693.406560 trunc from 0x028eb7 to 0x034e74 (0xbfbe bytes) 000849 1786402693.420940 mapread 0x021bc5 thru 0x02efdf (0xd41b bytes) 000850 1786402693.453452 mapwrite 0x0008d3 thru 0x0029c8 (0x20f6 bytes) 000851 1786402693.487875 trunc from 0x03c226 to 0x031fe1 (0xa244 bytes) 000852 1786402693.510623 trunc from 0x031fe2 to 0x002c07 (0x2f3da bytes) 000853 1786402693.537149 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2c08): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c08): Mirror 2: 0x12267976 (stale) Mirror 4: 0x12267976 (stale) CRC-32 checksum value for chunk [0, 0x2c08): Mirror 1: 0xf101c7a2 Mirror 5: 0xf101c7a2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c08: skipped 000854 1786402694.461136 mapwrite 0x010e1a thru 0x01a02b (0x9212 bytes) 000855 1786402694.631451 read 0x002af1 thru 0x00c053 (0x9563 bytes) 000856 1786402694.643400 read 0x00125d thru 0x0071a9 (0x5f4d bytes) 000857 1786402694.650636 trunc from 0x015955 to 0x018300 (0x29ac bytes) 000858 1786402694.655658 mapwrite 0x01889a thru 0x02839f (0xfb06 bytes) 000859 1786402694.711648 read 0x025d31 thru 0x02839f (0x266f bytes) 000860 1786402694.712468 write 0x03c353 thru 0x03ffff (0x3cad bytes) 000861 1786402694.713780 write 0x017b71 thru 0x01efb9 (0x7449 bytes) 000863 1786402694.725220 read 0x03d0d6 thru 0x03ffff (0x2f2a bytes) 000864 1786402694.726346 write 0x029123 thru 0x03507b (0xbf59 bytes) 000865 1786402694.728804 read 0x01f1ff thru 0x023227 (0x4029 bytes) 000866 1786402694.729801 punch from 0x014d0a to 0x01658b (0x1882 bytes) 000867 1786402694.735411 punch from 0x032794 to 0x03b3ce (0x8c3b bytes) 000868 1786402694.753385 write 0x0395e5 thru 0x03d285 (0x3ca1 bytes) 000870 1786402694.767799 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xfbc9fe67 (stale) Mirror 4: 0xe20eea22 (stale) Mirror 5: 0x8e66e952 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x28f2cebc Mirror 6: 0x28f2cebc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000871 1786402695.784689 mapread 0x005416 thru 0x00d9d5 (0x85c0 bytes) 000872 1786402695.826503 write 0x00e82d thru 0x01c791 (0xdf65 bytes) 000873 1786402695.891086 write 0x0301af thru 0x036456 (0x62a8 bytes) 000874 1786402695.903624 read 0x02fdbd thru 0x03e341 (0xe585 bytes) 000875 1786402695.914274 read 0x03e637 thru 0x03ffff (0x19c9 bytes) 000876 1786402695.925627 fallocate 0x02ebde thru 0x034ca0 (0x60c3 bytes) 000877 1786402695.932427 punch from 0x00b9d7 to 0x00c3dc (0xa06 bytes) 000879 1786402695.938065 fallocate 0x036429 thru 0x039504 (0x30dc bytes) 000880 1786402695.948869 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xfbc9fe67 (stale) Mirror 4: 0xe20eea22 (stale) Mirror 6: 0x28f2cebc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xfbc9fe67 (stale) Mirror 4: 0xe20eea22 (stale) Mirror 6: 0x28f2cebc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000881 1786402696.399503 trunc from 0x02c72f to 0x03555e (0x8e30 bytes) 000882 1786402696.407486 mapread 0x0384e6 thru 0x03ffff (0x7b1a bytes) 000883 1786402696.435227 punch from 0x037363 to 0x03ffff (0x8c9d bytes) 000884 1786402696.441351 trunc from 0x040000 to 0x0375ea (0x8a15 bytes) 000885 1786402696.469565 trunc from 0x00446e to 0x00b21b (0x6dae bytes) 000886 1786402696.476431 punch from 0x00ec9c to 0x0197aa (0xab0f bytes) 000887 1786402696.489111 trunc from 0x02dcaf to 0x03040d (0x275f bytes) 000889 1786402696.495625 punch from 0x0110d8 to 0x01da26 (0xc94f bytes) 000890 1786402696.507513 mapwrite 0x0111cf thru 0x01c7c5 (0xb5f7 bytes) 000891 1786402696.578489 trunc from 0x0375eb to 0x01d09b (0x1a54f bytes) 000892 1786402696.597507 read 0x002f84 thru 0x00f1f3 (0xc270 bytes) 000893 1786402696.609944 read 0x002ec5 thru 0x00fd82 (0xcebe bytes) 000894 1786402696.611436 trunc from 0x016b75 to 0x01d09b (0x6527 bytes) 000895 1786402696.619595 mapwrite 0x0254f8 thru 0x02af03 (0x5a0c bytes) 000896 1786402696.673472 fallocate 0x01c3c4 thru 0x01e431 (0x206e bytes) 000897 1786402696.680558 write 0x034d40 thru 0x03b7b5 (0x6a76 bytes) 000898 1786402696.683165 trunc from 0x0221ca to 0x02b4f6 (0x932d bytes) 000899 1786402696.689522 read 0x01f039 thru 0x022976 (0x393e bytes) 000900 1786402696.704737 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b7b6): Mirror 4: 0x4883c263 (stale) Mirror 6: 0x7112bdf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b7b6): Mirror 4: 0x4883c263 (stale) Mirror 6: 0x7112bdf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b7b6: skipped 000901 1786402696.994670 read 0x030dc2 thru 0x035593 (0x47d2 bytes) 000902 1786402697.007804 trunc from 0x03b7b6 to 0x026cba (0x14afb bytes) 000903 1786402697.037850 write 0x01536a thru 0x01dfac (0x8c43 bytes) 000904 1786402697.057301 fallocate 0x01c12a thru 0x02968b (0xd562 bytes) 000905 1786402697.070580 write 0x02a107 thru 0x038dfe (0xecf8 bytes) 000906 1786402697.072350 read 0x02131d thru 0x030049 (0xed2d bytes) 000908 1786402697.083603 mapread 0x0011bf thru 0x00a3d1 (0x9213 bytes) 000909 1786402697.086876 trunc from 0x03322b to 0x038dfe (0x5bd4 bytes) 000910 1786402697.102371 trunc from 0x038dff to 0x029178 (0xfc86 bytes) 000911 1786402697.124233 write 0x02363d thru 0x02a360 (0x6d24 bytes) 000912 1786402697.126602 punch from 0x010977 to 0x01283b (0x1ec5 bytes) 000914 1786402697.138135 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a361): Mirror 6: 0x73f786a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a361): Mirror 6: 0x73f786a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a361: skipped 000916 1786402697.609331 read 0x007058 thru 0x01393d (0xc8e6 bytes) 000917 1786402697.630454 write 0x029ae1 thru 0x0362ed (0xc80d bytes) 000919 1786402697.658330 mapread 0x02914f thru 0x0362ed (0xd19f bytes) 000920 1786402697.661657 trunc from 0x0362ee to 0x03cbe1 (0x68f4 bytes) 000921 1786402697.685854 fallocate 0x034fcf thru 0x03ffff (0xb031 bytes) 000922 1786402697.698480 trunc from 0x040000 to 0x001a98 (0x3e567 bytes) 000923 1786402697.746429 read 0x000426 thru 0x001a98 (0x1673 bytes) 000924 1786402697.765618 fallocate 0x000504 thru 0x00ef62 (0xea5f bytes) 000925 1786402697.776571 fallocate 0x034ee5 thru 0x0374eb (0x2607 bytes) 000927 1786402697.789751 fallocate 0x034be4 thru 0x0383db (0x37f8 bytes) 000928 1786402697.796766 mapwrite 0x02e25e thru 0x03c9e5 (0xe788 bytes) 000929 1786402697.906516 punch from 0x02e10f to 0x03260d (0x44ff bytes) 000930 1786402697.916440 fallocate 0x03f72e thru 0x03ffff (0x8d2 bytes) 000931 1786402697.922622 mapwrite 0x000289 thru 0x003d96 (0x3b0e bytes) 000932 1786402697.948215 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18e:0x0]: Invalid argument (22) 000933 1786402698.106944 mapread 0x01bf8a thru 0x02ba2e (0xfaa5 bytes) 000934 1786402698.157363 mapwrite 0x03105d thru 0x03d591 (0xc535 bytes) 000935 1786402698.216983 trunc from 0x025332 to 0x025bc1 (0x890 bytes) 000937 1786402698.224564 write 0x03251d thru 0x03e9c7 (0xc4ab bytes) 000938 1786402698.226580 punch from 0x02cedc to 0x032ab0 (0x5bd5 bytes) 000940 1786402698.246393 fallocate 0x0333db thru 0x03ffff (0xcc25 bytes) 000941 1786402698.252497 read 0x023593 thru 0x02b212 (0x7c80 bytes) 000942 1786402698.259200 write 0x00771e thru 0x00d0d7 (0x59ba bytes) 000943 1786402698.270195 trunc from 0x0380e4 to 0x03e9c7 (0x68e4 bytes) 000944 1786402698.276313 mapread 0x02a6c6 thru 0x035629 (0xaf64 bytes) 000945 1786402698.291234 write 0x0009f3 thru 0x008dc3 (0x83d1 bytes) 000946 1786402698.298064 trunc from 0x00a1a8 to 0x012aff (0x8958 bytes) 000947 1786402698.315436 fallocate 0x008f20 thru 0x00d5bc (0x469d bytes) 000948 1786402698.321197 mapread 0x020088 thru 0x021524 (0x149d bytes) 000949 1786402698.323218 punch from 0x030807 to 0x03a4bd (0x9cb7 bytes) 000950 1786402698.328501 mapread 0x01fd4c thru 0x022537 (0x27ec bytes) 000951 1786402698.330490 read 0x015039 thru 0x022418 (0xd3e0 bytes) 000952 1786402698.340828 mapwrite 0x03df05 thru 0x03ffff (0x20fb bytes) 000953 1786402698.387497 MIRROR_EXTEND to 2 mirrors 000954 1786402698.983946 trunc from 0x040000 to 0x01d206 (0x22df9 bytes) 000955 1786402699.072880 trunc from 0x003be3 to 0x0069cd (0x2deb bytes) 000956 1786402699.079583 punch from 0x00a537 to 0x00d733 (0x31fd bytes) 000958 1786402699.086852 read 0x00dc0d thru 0x013373 (0x5767 bytes) 000959 1786402699.097555 mapwrite 0x02c44f thru 0x03b537 (0xf0e9 bytes) 000960 1786402699.199060 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18e:0x0]: Invalid argument (22) 000961 1786402699.360910 mapread 0x009659 thru 0x00f64c (0x5ff4 bytes) 000962 1786402699.387408 mapwrite 0x01b796 thru 0x02b685 (0xfef0 bytes) 000963 1786402699.423053 punch from 0x00bfed to 0x01b582 (0xf596 bytes) 000965 1786402699.427343 trunc from 0x03b538 to 0x027561 (0x13fd6 bytes) 000966 1786402699.444737 trunc from 0x027562 to 0x013996 (0x13bcb bytes) 000967 1786402699.460172 mapwrite 0x037e35 thru 0x03f278 (0x7444 bytes) 000969 1786402699.526918 mapwrite 0x019c7c thru 0x023240 (0x95c5 bytes) 000970 1786402699.568759 write 0x02861d thru 0x033836 (0xb21a bytes) 000971 1786402699.570685 trunc from 0x03f279 to 0x010bd6 (0x2e6a2 bytes) 000972 1786402699.597380 trunc from 0x005cd4 to 0x007118 (0x1445 bytes) 000973 1786402699.611577 punch from 0x003297 to 0x006cdd (0x3a47 bytes) 000975 1786402699.623122 punch from 0x00b77e to 0x010bd6 (0x5459 bytes) 000977 1786402699.632113 write 0x014711 thru 0x01e7df (0xa0cf bytes) 000978 1786402699.636212 mapwrite 0x03ed6e thru 0x03ffff (0x1292 bytes) 000979 1786402699.705946 mapread 0x0162f6 thru 0x018a51 (0x275c bytes) 000980 1786402699.708472 punch from 0x02672c to 0x029f78 (0x384d bytes) 000981 1786402699.721987 read 0x02aab7 thru 0x02e9f3 (0x3f3d bytes) 000982 1786402699.740211 MIRROR_EXTEND to 2 mirrors 000983 1786402701.020089 punch from 0x020bd1 to 0x02dd0e (0xd13e bytes) 000984 1786402701.107300 mapread 0x00921b thru 0x0120fb (0x8ee1 bytes) 000985 1786402701.153092 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18e:0x0]: Invalid argument (22) 000986 1786402701.336970 mapwrite 0x02ca0e thru 0x035ab9 (0x90ac bytes) 000988 1786402701.383457 fallocate 0x03cf31 thru 0x03ffff (0x30cf bytes) 000989 1786402701.389118 punch from 0x0383cd to 0x03db15 (0x5749 bytes) 000990 1786402701.395084 trunc from 0x040000 to 0x00fe37 (0x301c8 bytes) 000991 1786402701.417194 mapwrite 0x021c4c thru 0x023e8d (0x2242 bytes) 000992 1786402701.456060 trunc from 0x019e28 to 0x023e8d (0xa066 bytes) 000994 1786402701.462845 mapread 0x014888 thru 0x019374 (0x4aed bytes) 000995 1786402701.484663 mapread 0x015ffe thru 0x0217f1 (0xb7f4 bytes) 000996 1786402701.495472 write 0x00684c thru 0x006de5 (0x59a bytes) 000997 1786402701.501490 punch from 0x00ad46 to 0x015263 (0xa51e bytes) 000998 1786402701.507876 fallocate 0x00f38f thru 0x0123f8 (0x306a bytes) 000999 1786402701.513553 trunc from 0x023e8e to 0x024766 (0x8d9 bytes) 001000 1786402701.532239 trunc from 0x020bf0 to 0x024766 (0x3b77 bytes) All operations completed A-OK! PASS 70a (86s) == sanity-flr test 71: check mirror extend parent fid ==== 18:58:28 (1786402708) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x1cc:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (13s) == sanity-flr test 72: check append on FLR file ========== 18:58:41 (1786402721) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xf5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.290663 s, 7.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xf5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10969 s, 9.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xf6:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xf5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xf6:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xf5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x119:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 18:58:52 (1786402732) Starting client: oleg235-client.virtnet: -o user_xattr,flock 192.168.202.135@tcp:/lustre /mnt/lustre2 Starting client: oleg235-client.virtnet: -o user_xattr,flock 192.168.202.135@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9271168 .. Extending file size to 5774080 .. Extending file size to 7417184 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8273248 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61103 61104 61107 61108 61109 done 192.168.202.135@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg235-client.virtnet /mnt/lustre2 (opts:) 192.168.202.135@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg235-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (94s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 19:00:27 (1786402827) Starting client: oleg235-client.virtnet: -o user_xattr,flock 192.168.202.135@tcp:/lustre /mnt/lustre2 Starting client: oleg235-client.virtnet: -o user_xattr,flock 192.168.202.135@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8661696 .. Extending file size to 4190592 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7836576 .. Extending file size to 8132960 .. Extending file size to 8333504 .. Extending file size to 8430752 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8726080 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9042400 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 62528 62529 62532 62533 62537 failed 192.168.202.135@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg235-client.virtnet /mnt/lustre2 (opts:) 192.168.202.135@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg235-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (83s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 19:01:50 (1786402910) Starting client: oleg235-client.virtnet: -o user_xattr,flock 192.168.202.135@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2244 1269060 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2144 1269160 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 16692 3590328 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24452 3582568 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 41144 7172896 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2244 1269060 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2144 1269160 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 16692 3590328 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 24452 3582568 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 41144 7172896 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.104899 s, 20.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.267815 s, 7.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xa2ae03e1 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.135@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg235-client.virtnet /mnt/lustre2 (opts:) PASS 200c (21s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 19:02:12 (1786402932) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.123087 s, 17.0 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 19:02:20 (1786402940) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x63:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10c:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10c:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10d:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 19:02:30 (1786402950) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.143064 s, 14.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x22e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x22f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x230:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 19:02:38 (1786402958) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.110521 s, 19.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } PASS 204b (11s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 19:02:49 (1786402969) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.106656 s, 19.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x22f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x231:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x22f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x231:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x232:0x0] } PASS 204c (36s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 19:03:25 (1786403005) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg235-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177023 s, 5.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x230:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x234:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x233:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x230:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x234:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x233:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x231:0x0] } PASS 204d (58s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 19:04:23 (1786403063) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg235-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x110:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x111:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.160738 s, 13.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x110:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x111:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x110:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x111:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (59s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 19:05:22 (1786403122) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg235-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg235-server: Pool lustre.test_204f created Adding targets to pool oleg235-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg235-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.129414 s, 16.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x112:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x112:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg235-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg235-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg235-server: Pool lustre.test_204f destroyed PASS 204f (80s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 19:06:42 (1786403202) PASS 205a (10s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 19:06:52 (1786403212) PASS 205b (10s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 19:07:02 (1786403222) oleg235-server: Pool lustre.test_206 created Adding targets to pool oleg235-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg235-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg235-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg235-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg235-server: Pool lustre.test_206 destroyed PASS 206 (30s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 19:07:32 (1786403252) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962795 s, 10.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529094 s, 19.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xe927f52e Mirror 3: 0xe927f52e PASS 207 (15s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 19:07:48 (1786403268) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (6s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 19:07:54 (1786403274) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 19:07:58 (1786403278) Starting client: oleg235-client.virtnet: -o user_xattr,flock 192.168.202.135@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0354109 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1460000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xb90000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xb90000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xb90000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xd40000 Read this (4 bytes): BBBB 192.168.202.135@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg235-client.virtnet /mnt/lustre2 (opts:) PASS 209a (43s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 19:08:42 (1786403322) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0313834 s, 261 kB/s the very first read read with warm cache PASS 209b (10s) == sanity-flr test 210a: handle broken mirrored lovea ==== 19:08:52 (1786403332) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421214 s, 24.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x82 lmm_fid: [0x200000405:0x82:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x82 lmm_fid: [0x200000405:0x82:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13d:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611266 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-10 19:08:54.000000000 -0400 Modify: 2026-08-10 19:08:54.000000000 -0400 Change: 2026-08-10 19:08:56.000000000 -0400 Birth: 2026-08-10 19:08:53.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 19:09:03 (1786403343) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593053 s, 17.7 MB/s oleg235-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg235-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg235-server: O/280000401/d28/11c: File not found by ext2_lookup PASS 210b (24s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 19:09:28 (1786403368) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.244976 s, 167 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0307526 s, 133 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 10 19:09 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x87:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 10 19:09 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 19:09:36 (1786403376) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 19:09:46 (1786403386) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1910 sec =========== 19:09:55 (1786403395) === sanity-flr: start cleanup 19:09:57 (1786403397) === === sanity-flr: finish cleanup 19:10:06 (1786403406) ===