-----============= acceptance-small: sanity-flr ============----- Mon Apr 15 15:10:41 EDT 2024 excepting tests: 6 201 44c 49a === sanity-flr: start setup 15:10:45 (1713208245) === oleg414-client.virtnet: executing check_config_client /mnt/lustre oleg414-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg414-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b6095800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b6095800.idle_timeout=debug disable quota as required oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 15:10:52 (1713208252) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f7649] debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0a: lfs mirror create with -N option == 15:10:53 (1713208253) PASS 0a (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 15:10:58 (1713208258) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 15:11:00 (1713208260) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0d: lfs mirror extend with -N option == 15:11:02 (1713208262) PASS 0d (5s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 15:11:08 (1713208268) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 15:11:11 (1713208271) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0g: lfs mirror create flags support === 15:11:13 (1713208273) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b6095800.stats=clear osc.lustre-OST0001-osc-ffff8800b6095800.stats=clear PASS 0g (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 15:11:16 (1713208276) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 15:11:20 (1713208280) PASS 0j (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 15:11:24 (1713208284) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 1: create components with setstripe options ========================================================== 15:11:26 (1713208286) lfs mirror mirror: cannot get UNLOCK lease, ext 4: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 2: create components from existing files ========================================================== 15:11:30 (1713208290) PASS 2 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 3: create components from files located on different MDTs ========================================================== 15:11:34 (1713208294) PASS 3 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 15:11:37 (1713208297) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 15:11:41 (1713208301) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 7: Create mirror with DoM component === 15:11:45 (1713208305) PASS 7 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 15:11:48 (1713208308) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB) copied, 0.183765 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0375212 s, 27.9 MB/s PASS 21 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 15:11:52 (1713208312) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.0456942 s, 184 MB/s PASS 22 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 15:11:55 (1713208315) Stopping /mnt/lustre-ost1 (opts:) on oleg414-server oleg414-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6095800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6095800.ost_server_uuid in DISCONN state after 1 sec Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg414-server: oleg414-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg414-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg414-server oleg414-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6095800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6095800.ost_server_uuid in DISCONN state after 3 sec Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg414-server: oleg414-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg414-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (23s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 15:12:20 (1713208340) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0336066 s, 62.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg414-server oleg414-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6095800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6095800.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg414-server: oleg414-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg414-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (10s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33a: read can choose available mirror to read ========================================================== 15:12:31 (1713208351) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg414-server oleg414-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6095800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6095800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg414-server: oleg414-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg414-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg414-server oleg414-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6095800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6095800.ost_server_uuid in DISCONN state after 4 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg414-server: oleg414-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg414-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (18s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 15:12:51 (1713208371) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.724876 s, 72.3 MB/s -rw-r--r-- 1 root root 52428800 Apr 15 15:12 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.182s user 0m0.000s sys 0m0.101s Stopping /mnt/lustre-ost1 (opts:) on oleg414-server oleg414-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6095800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6095800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.201s user 0m0.000s sys 0m0.118s Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg414-server: oleg414-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg414-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg414-server oleg414-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6095800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6095800.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost1)... real 0m0.215s user 0m0.000s sys 0m0.113s Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg414-server: oleg414-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg414-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg414-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b6095800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6095800.ost_server_uuid in FULL state after 0 sec PASS 33b (18s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 15:13:11 (1713208391) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 15:13:13 (1713208393) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 15:13:16 (1713208396) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 35: allow to write to mirrored files == 15:13:18 (1713208398) PASS 35 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36a: write to mirrored files ========== 15:13:21 (1713208401) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.110077 s, 114 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 15:13:25 (1713208405) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00659184 s, 159 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00917839 s, 114 MB/s PASS 36b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 15:13:28 (1713208408) PASS 36c (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 15:13:32 (1713208412) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000578518 s, 1.8 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00053278 s, 1.9 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000551816 s, 1.9 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000557811 s, 1.8 MB/s flv=10 olv=10 PASS 36d (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 37: mirror I/O API verification ======= 15:13:35 (1713208415) create test files with size 15735901 .. 665aed36020440616a24fde2c90540e7 - ecc4d9b029aad0a92e049e20cd87c4f9 - 99b335112c1e13e85bed1f18c0e7447d - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg414-client.virtnet /mnt/lustre (opts:) Starting client: oleg414-client.virtnet: -o user_xattr,flock oleg414-server@tcp:/lustre /mnt/lustre PASS 37 (5s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 38: resync ============================ 15:13:41 (1713208421) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 15:13:46 (1713208426) PASS 39 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 15:13:49 (1713208429) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0176215 s, 59.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3c:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3d:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3a:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00763963 s, 137 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3b:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3c:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3e:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 41: lfs mirror resync check =========== 15:13:52 (1713208432) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0239521 s, 87.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0906527 s, 46.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.036716 s, 57.1 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 42: lfs mirror verify ================= 15:13:56 (1713208436) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 43a: mirror pick on write ============= 15:13:58 (1713208438) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 15:14:00 (1713208440) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00634358 s, 165 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44a: lfs mirror split check =========== 15:14:03 (1713208443) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0695273 s, 45.2 MB/s lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) PASS 44a (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44b: mirror split does not create stale file ========================================================== 15:14:07 (1713208447) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x47:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x46:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (2s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 15:14:11 (1713208451) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0566165 s, 185 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg414-client.virtnet /mnt/lustre (opts:) Starting client: oleg414-client.virtnet: -o user_xattr,flock oleg414-server@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 15:14:15 (1713208455) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8800b586a000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0619574 s, 169 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg414-client.virtnet /mnt/lustre (opts:) Starting client: oleg414-client.virtnet: -o user_xattr,flock oleg414-server@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff8800ae434800.xattr_cache=1 PASS 44e (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 15:14:19 (1713208459) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (7s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 46: Verify setstripe --copy option ==== 15:14:27 (1713208467) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (6s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 47: Verify mirror obj alloc =========== 15:14:35 (1713208475) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 48: Verify snapshot mirror ============ 15:14:37 (1713208477) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0689144 s, 45.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0474479 s, 66.3 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 537547549f8cc85c5b412d17b04e9214 - mirror 1 checksum: e5bd70669aa7a544757a13ede0839172 - mirror 2 checksum: 537547549f8cc85c5b412d17b04e9214 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Apr 15 15:14:38 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x54:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Apr 15 15:14:38 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x55:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 537547549f8cc85c5b412d17b04e9214 - mirror 1 checksum: e5bd70669aa7a544757a13ede0839172 - mirror 2 checksum: e5bd70669aa7a544757a13ede0839172 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Apr 15 15:14:38 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x54:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Apr 15 15:14:38 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x55:0x0] } PASS 48 (2s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50A: mirror split update layout generation ========================================================== 15:14:41 (1713208481) Failing mds1 on oleg414-server Stopping /mnt/lustre-mds1 (opts:) on oleg414-server 15:14:43 (1713208483) shut down Failover mds1 to oleg414-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0000 15:14:55 (1713208495) targets are mounted 15:14:55 (1713208495) facet_failover done oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00841006 s, 125 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xd lmm_fid: [0x200000405:0xd:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } PASS 50A (22s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 15:15:04 (1713208504) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50b: mirror rsync handles sparseness == 15:15:06 (1713208506) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 15:15:09 (1713208509) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.0022927 s, 7.1 MB/s PASS 50c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50d: mirror rsync keep holes ========== 15:15:12 (1713208512) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 15:15:14 (1713208514) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 15:15:16 (1713208516) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) PASS 60b (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 15:15:21 (1713208521) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (18s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 15:15:40 (1713208540) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 15:15:58 (1713208558) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 15:16:17 (1713208577) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 70: mirror create and split race ====== 15:16:20 (1713208580) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 28667 mirror split pid 28668 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7301: 28668 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4697: 28667 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) PASS 70 (61s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 70a: flr mode fsx test ================ 15:17:23 (1713208643) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 8643 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000004 1713208643.965886 write 0x005602 thru 0x00bc10 (0x660f bytes) 000007 1713208643.967654 read 0x00241d thru 0x00bc10 (0x97f4 bytes) 000009 1713208643.969616 read 0x001179 thru 0x00bc10 (0xaa98 bytes) 000010 1713208643.970881 read 0x00ad34 thru 0x00bc10 (0xedd bytes) 000013 1713208643.971568 read 0x00725e thru 0x00bc10 (0x49b3 bytes) 000015 1713208643.971986 mapwrite 0x01bb73 thru 0x0294b2 (0xd940 bytes) 000018 1713208644.008216 write 0x0344b0 thru 0x03a3d4 (0x5f25 bytes) 000019 1713208644.008868 read 0x008e45 thru 0x00c6ff (0x38bb bytes) 000020 1713208644.009129 mapread 0x032097 thru 0x03a3d4 (0x833e bytes) truncating to largest ever: 0x166fe 000021 1713208644.011925 trunc from 0x03a3d5 to 0x0166fd (0x23cd7 bytes) 000022 1713208644.015916 write 0x038717 thru 0x03ffff (0x78e9 bytes) 000023 1713208644.016889 mapwrite 0x00605c thru 0x0079df (0x1984 bytes) 000024 1713208644.031898 trunc from 0x040000 to 0x0117a3 (0x2e85c bytes) 000025 1713208644.035934 write 0x020b1a thru 0x025335 (0x481c bytes) 000026 1713208644.036544 read 0x00a6ee thru 0x019e61 (0xf774 bytes) 000027 1713208644.038582 write 0x0058a2 thru 0x010d86 (0xb4e5 bytes) 000028 1713208644.039358 mapread 0x024174 thru 0x025335 (0x11c2 bytes) 000029 1713208644.039929 mapwrite 0x03bfa7 thru 0x03ffff (0x4059 bytes) truncating to largest ever: 0x36f13 000032 1713208644.064520 trunc from 0x040000 to 0x036f12 (0x90ed bytes) truncating to largest ever: 0x37510 000035 1713208644.069659 trunc from 0x036f13 to 0x03750f (0x5fd bytes) 000036 1713208644.073164 read 0x003ec0 thru 0x0120d1 (0xe212 bytes) 000038 1713208644.074744 MIRROR_EXTEND to 2 mirrors 000039 1713208644.181813 trunc from 0x037510 to 0x02729b (0x10274 bytes) 000040 1713208644.200905 trunc from 0x02729c to 0x004b54 (0x22747 bytes) 000041 1713208644.205595 write 0x0173aa thru 0x023a2a (0xc681 bytes) 000043 1713208644.206701 write 0x0361df thru 0x03a0ef (0x3f11 bytes) 000044 1713208644.207294 read 0x014b1e thru 0x01fb20 (0xb003 bytes) 000047 1713208644.209439 mapread 0x02c556 thru 0x03963f (0xd0ea bytes) 000048 1713208644.213078 write 0x03812f thru 0x03ffff (0x7ed1 bytes) 000049 1713208644.213815 read 0x00458a thru 0x00b588 (0x6fff bytes) 000050 1713208644.215955 mapread 0x03d5a3 thru 0x03ffff (0x2a5d bytes) 000051 1713208644.216424 trunc from 0x040000 to 0x01e4fd (0x21b02 bytes) 000052 1713208644.219807 mapread 0x00d892 thru 0x0175e2 (0x9d51 bytes) 000053 1713208644.221224 MIRROR_EXTEND to 3 mirrors 000054 1713208644.336517 mapwrite 0x005401 thru 0x00c49d (0x709d bytes) 000056 1713208644.381857 mapread 0x00ebe3 thru 0x01cbd5 (0xdff3 bytes) 000057 1713208644.385574 mapwrite 0x0239b7 thru 0x023ffe (0x648 bytes) 000059 1713208644.408533 trunc from 0x023fff to 0x02518f (0x1191 bytes) 000060 1713208644.411903 write 0x038c11 thru 0x03ffff (0x73ef bytes) 000061 1713208644.412587 write 0x03f64a thru 0x03ffff (0x9b6 bytes) 000062 1713208644.413971 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000063 1713208644.462519 MIRROR_RESYNC 2 mirrors 000064 1713208644.549579 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000065 1713208644.577881 trunc from 0x040000 to 0x02b6fb (0x14904 bytes) 000066 1713208644.586939 mapread 0x008ed5 thru 0x00a4a5 (0x15d1 bytes) 000067 1713208644.591580 read 0x0106ee thru 0x01516d (0x4a80 bytes) 000068 1713208644.593661 trunc from 0x02b6fc to 0x01ed44 (0xc9b7 bytes) 000070 1713208644.597658 mapread 0x0085d3 thru 0x0102c8 (0x7cf6 bytes) 000071 1713208644.598346 write 0x03b610 thru 0x03ffff (0x49f0 bytes) 000072 1713208644.599189 read 0x0366b5 thru 0x03ffff (0x994b bytes) 000073 1713208644.601833 read 0x02ceb7 thru 0x02feeb (0x3035 bytes) 000078 1713208644.603846 mapread 0x029d4f thru 0x0379c5 (0xdc77 bytes) 000079 1713208644.604426 trunc from 0x040000 to 0x0243ff (0x1bc00 bytes) 000080 1713208644.608478 mapread 0x01ae5c thru 0x01c127 (0x12cc bytes) 000081 1713208644.608868 write 0x018214 thru 0x0186f8 (0x4e5 bytes) 000083 1713208644.609509 write 0x035259 thru 0x03a6d3 (0x547b bytes) 000084 1713208644.609965 read 0x02ca60 thru 0x0324c2 (0x5a63 bytes) 000086 1713208644.613518 MIRROR_EXTEND to 2 mirrors 000088 1713208644.717927 trunc from 0x03a6d4 to 0x00c4f0 (0x2e1e3 bytes) 000089 1713208644.737782 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000090 1713208644.767628 mapwrite 0x02708b thru 0x0323e0 (0xb356 bytes) 000092 1713208644.793050 trunc from 0x0323e1 to 0x027226 (0xb1ba bytes) 000093 1713208644.796364 write 0x015d0a thru 0x019caa (0x3fa1 bytes) 000094 1713208644.798793 trunc from 0x027227 to 0x00883c (0x1e9ea bytes) 000095 1713208644.802463 mapread 0x001c02 thru 0x00883c (0x6c3b bytes) 000096 1713208644.804729 trunc from 0x00883d to 0x010ddc (0x85a0 bytes) 000097 1713208644.808430 mapwrite 0x013804 thru 0x01d8f8 (0xa0f5 bytes) 000098 1713208644.831476 mapread 0x007102 thru 0x00cfd5 (0x5ed4 bytes) 000099 1713208644.833238 write 0x002ac7 thru 0x0066cf (0x3c09 bytes) 000104 1713208644.834771 mapwrite 0x030bf0 thru 0x03a305 (0x9716 bytes) 000105 1713208644.856618 mapread 0x016a4e thru 0x01ec69 (0x821c bytes) 000106 1713208644.858836 mapread 0x007899 thru 0x00f55c (0x7cc4 bytes) 000107 1713208644.859269 trunc from 0x03a306 to 0x002e09 (0x374fc bytes) 000109 1713208644.863440 read 0x000824 thru 0x002e09 (0x25e6 bytes) 000110 1713208644.864843 mapwrite 0x00d1e9 thru 0x0138a2 (0x66ba bytes) 000111 1713208644.883851 trunc from 0x0138a3 to 0x021fc9 (0xe727 bytes) 000112 1713208644.887417 write 0x019fb4 thru 0x01b9e0 (0x1a2d bytes) 000114 1713208644.890108 mapread 0x01d092 thru 0x021fc9 (0x4f38 bytes) 000115 1713208644.890584 trunc from 0x021fca to 0x009954 (0x18675 bytes) 000118 1713208644.894548 mapread 0x0036e1 thru 0x009954 (0x6274 bytes) 000119 1713208644.896085 read 0x00096d thru 0x009954 (0x8fe8 bytes) 000122 1713208644.896859 mapread 0x00423e thru 0x009954 (0x5717 bytes) 000123 1713208644.897206 trunc from 0x009955 to 0x0126c3 (0x8d6f bytes) 000128 1713208644.901644 mapwrite 0x00542b thru 0x013848 (0xe41e bytes) 000130 1713208644.922465 mapread 0x011949 thru 0x013848 (0x1f00 bytes) 000131 1713208644.922927 mapread 0x004683 thru 0x00cc3b (0x85b9 bytes) 000134 1713208644.923790 mapwrite 0x0051d5 thru 0x00f84f (0xa67b bytes) 000135 1713208644.937055 trunc from 0x013849 to 0x029035 (0x157ed bytes) 000140 1713208644.941535 mapread 0x01eb1f thru 0x026e5b (0x833d bytes) 000145 1713208644.945450 mapread 0x00737f thru 0x0150df (0xdd61 bytes) 000148 1713208644.946731 read 0x01bddf thru 0x01eace (0x2cf0 bytes) 000149 1713208644.947021 mapwrite 0x03a713 thru 0x03ffff (0x58ed bytes) 000150 1713208644.968028 mapwrite 0x012101 thru 0x01d704 (0xb604 bytes) 000151 1713208644.983275 read 0x038f32 thru 0x03ffff (0x70ce bytes) 000152 1713208644.983826 mapwrite 0x00e778 thru 0x014a1a (0x62a3 bytes) 000154 1713208644.997388 write 0x00e610 thru 0x011acb (0x34bc bytes) 000155 1713208644.997899 write 0x014941 thru 0x022ebd (0xe57d bytes) 000156 1713208644.998580 mapwrite 0x01c8f3 thru 0x02154b (0x4c59 bytes) 000158 1713208645.013795 write 0x03a9a0 thru 0x03ffff (0x5660 bytes) 000160 1713208645.014978 read 0x02bce5 thru 0x03babd (0xfdd9 bytes) truncating to largest ever: 0x3b26d 000162 1713208645.015600 trunc from 0x040000 to 0x03b26c (0x4d93 bytes) 000168 1713208645.021009 read 0x0366a8 thru 0x03b26c (0x4bc5 bytes) 000170 1713208645.021691 mapwrite 0x01ac80 thru 0x028e63 (0xe1e4 bytes) 000172 1713208645.038464 mapwrite 0x0268e8 thru 0x027f9c (0x16b5 bytes) 000175 1713208645.052477 mapwrite 0x02dc36 thru 0x0385a9 (0xa974 bytes) 000176 1713208645.067815 read 0x026dd2 thru 0x02959d (0x27cc bytes) 000179 1713208645.068518 mapread 0x006728 thru 0x00e39c (0x7c75 bytes) 000180 1713208645.069042 mapread 0x01dea0 thru 0x02a0e4 (0xc245 bytes) 000185 1713208645.071020 mapread 0x030bcb thru 0x03159b (0x9d1 bytes) 000190 1713208645.074104 MIRROR_EXTEND to 2 mirrors 000193 1713208645.181099 mapread 0x024021 thru 0x026f20 (0x2f00 bytes) 000196 1713208645.187852 mapread 0x024346 thru 0x0332e9 (0xefa4 bytes) 000201 1713208645.191879 MIRROR_EXTEND to 3 mirrors 000202 1713208645.306946 read 0x0107f8 thru 0x012f84 (0x278d bytes) 000204 1713208645.308931 read 0x01b58a thru 0x020adb (0x5552 bytes) truncating to largest ever: 0x3ef3c 000205 1713208645.312152 trunc from 0x03b26d to 0x03ef3b (0x3ccf bytes) 000206 1713208645.328837 mapwrite 0x001876 thru 0x010da9 (0xf534 bytes) 000207 1713208645.359326 trunc from 0x03ef3c to 0x0204ca (0x1ea71 bytes) 000213 1713208645.365276 trunc from 0x0204cb to 0x002dad (0x1d71d bytes) 000214 1713208645.368573 write 0x02e9ae thru 0x039544 (0xab97 bytes) 000215 1713208645.369638 write 0x0392b9 thru 0x03edd9 (0x5b21 bytes) 000218 1713208645.371001 mapwrite 0x031c1d thru 0x03a554 (0x8938 bytes) 000220 1713208645.385428 mapread 0x019390 thru 0x028a70 (0xf6e1 bytes) 000221 1713208645.402282 read 0x01bbb9 thru 0x01d477 (0x18bf bytes) 000222 1713208645.402609 read 0x011351 thru 0x01e336 (0xcfe6 bytes) 000224 1713208645.405783 read 0x0047d6 thru 0x00b3b2 (0x6bdd bytes) 000226 1713208645.406251 read 0x03522f thru 0x03edd9 (0x9bab bytes) 000230 1713208645.408414 MIRROR_EXTEND to 4 mirrors 000231 1713208645.510569 write 0x00104e thru 0x006f3c (0x5eef bytes) 000232 1713208645.527573 trunc from 0x03edda to 0x0121ca (0x2cc0f bytes) 000237 1713208645.536928 write 0x0170c5 thru 0x018f52 (0x1e8e bytes) 000239 1713208645.537990 mapwrite 0x00fcc4 thru 0x010009 (0x346 bytes) 000241 1713208645.554289 mapread 0x00a1be thru 0x015e04 (0xbc47 bytes) 000243 1713208645.555000 write 0x00ef18 thru 0x00f2c0 (0x3a9 bytes) 000246 1713208645.555945 mapread 0x012b10 thru 0x0154e6 (0x29d7 bytes) 000248 1713208645.556567 mapwrite 0x009f97 thru 0x0164d0 (0xc53a bytes) 000249 1713208645.570664 mapread 0x0135ec thru 0x018f52 (0x5967 bytes) 000250 1713208645.571080 write 0x03fd9e thru 0x03ffff (0x262 bytes) 000252 1713208645.571937 read 0x00fab3 thru 0x01566a (0x5bb8 bytes) 000253 1713208645.572199 mapwrite 0x011bfe thru 0x021656 (0xfa59 bytes) 000254 1713208645.588924 mapread 0x035d90 thru 0x03ffff (0xa270 bytes) 000255 1713208645.593667 read 0x007a4b thru 0x00818d (0x743 bytes) 000257 1713208645.596202 read 0x02b78b thru 0x02f4b9 (0x3d2f bytes) 000258 1713208645.596506 mapread 0x02dc42 thru 0x02ec3e (0xffd bytes) 000261 1713208645.597602 trunc from 0x040000 to 0x007998 (0x38667 bytes) 000263 1713208645.602327 trunc from 0x007999 to 0x026552 (0x1ebba bytes) 000264 1713208645.605804 mapwrite 0x02a4f0 thru 0x0372ba (0xcdcb bytes) 000265 1713208645.629015 mapread 0x02680a thru 0x02d967 (0x715e bytes) 000266 1713208645.629747 trunc from 0x0372bb to 0x01ab12 (0x1c7a8 bytes) 000267 1713208645.634513 MIRROR_RESYNC 4 mirrors 000268 1713208645.759605 trunc from 0x01ab13 to 0x03588b (0x1ad79 bytes) 000270 1713208645.779287 mapread 0x029e07 thru 0x03588b (0xba85 bytes) 000271 1713208645.785632 write 0x02469b thru 0x02725d (0x2bc3 bytes) 000272 1713208645.787234 MIRROR_EXTEND to 5 mirrors 000273 1713208645.892546 mapwrite 0x021c81 thru 0x02f0dd (0xd45d bytes) 000274 1713208645.940119 trunc from 0x03588c to 0x0133aa (0x224e1 bytes) 000275 1713208645.948562 trunc from 0x0133ab to 0x0187e4 (0x543a bytes) 000277 1713208645.952431 read 0x014022 thru 0x0187e4 (0x47c3 bytes) 000278 1713208645.954779 mapread 0x0022af thru 0x005042 (0x2d94 bytes) 000281 1713208645.957760 mapwrite 0x003686 thru 0x00e11e (0xaa99 bytes) 000285 1713208645.974219 write 0x00affa thru 0x01801a (0xd021 bytes) 000286 1713208645.976497 MIRROR_RESYNC 5 mirrors 000288 1713208646.130079 trunc from 0x0187e5 to 0x0150c8 (0x371c bytes) 000289 1713208646.143760 mapread 0x0036a7 thru 0x003c4d (0x5a7 bytes) 000290 1713208646.145210 read 0x011e7a thru 0x0150c8 (0x324f bytes) 000291 1713208646.147211 mapwrite 0x02caa7 thru 0x03457b (0x7ad5 bytes) 000292 1713208646.171351 mapread 0x0015b7 thru 0x006320 (0x4d6a bytes) 000293 1713208646.173861 mapread 0x01143a thru 0x012be0 (0x17a7 bytes) 000294 1713208646.174289 mapread 0x02c39f thru 0x02d3cc (0x102e bytes) 000295 1713208646.174846 read 0x017336 thru 0x0267e6 (0xf4b1 bytes) 000296 1713208646.177345 write 0x03cd65 thru 0x03d186 (0x422 bytes) 000297 1713208646.177958 read 0x01d48b thru 0x02cb28 (0xf69e bytes) 000298 1713208646.178336 write 0x028ab7 thru 0x02bf0b (0x3455 bytes) 000299 1713208646.178925 mapread 0x015f2c thru 0x021156 (0xb22b bytes) 000300 1713208646.179556 mapwrite 0x013798 thru 0x01a1cc (0x6a35 bytes) 000301 1713208646.194032 read 0x010d40 thru 0x01cc5a (0xbf1b bytes) 000302 1713208646.194555 write 0x01be77 thru 0x026cee (0xae78 bytes) 000303 1713208646.195276 read 0x03362b thru 0x03d186 (0x9b5c bytes) 000304 1713208646.199017 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000305 1713208646.243165 trunc from 0x03d187 to 0x008016 (0x35170 bytes) 000306 1713208646.246326 mapwrite 0x019b54 thru 0x023b36 (0x9fe3 bytes) 000307 1713208646.269732 MIRROR_RESYNC 4 mirrors 000308 1713208646.395771 mapread 0x00c852 thru 0x00d1ae (0x95d bytes) 000310 1713208646.398510 mapread 0x00f6c5 thru 0x0117f7 (0x2133 bytes) 000312 1713208646.401082 mapwrite 0x028356 thru 0x0344cf (0xc17a bytes) 000314 1713208646.441643 trunc from 0x0344d0 to 0x00779e (0x2cd31 bytes) 000316 1713208646.446757 trunc from 0x00779f to 0x030ab6 (0x29318 bytes) 000317 1713208646.450124 trunc from 0x030ab7 to 0x031425 (0x96f bytes) 000318 1713208646.453699 read 0x0237f2 thru 0x02bab7 (0x82c6 bytes) 000320 1713208646.456688 trunc from 0x031426 to 0x025875 (0xbbb0 bytes) 000321 1713208646.460649 write 0x00a3b1 thru 0x00a6eb (0x33b bytes) 000323 1713208646.463331 write 0x0117b7 thru 0x0183dd (0x6c27 bytes) 000324 1713208646.465160 write 0x030375 thru 0x03c3fa (0xc086 bytes) 000325 1713208646.467160 MIRROR_RESYNC 4 mirrors 000327 1713208646.595709 mapwrite 0x036bfa thru 0x037503 (0x90a bytes) 000329 1713208646.636632 trunc from 0x03c3fb to 0x00cf26 (0x2f4d4 bytes) 000332 1713208646.645901 trunc from 0x00cf27 to 0x00f790 (0x286a bytes) 000333 1713208646.649653 trunc from 0x00f791 to 0x01682a (0x709a bytes) 000335 1713208646.653559 mapwrite 0x01122a thru 0x01f1d3 (0xdfaa bytes) 000336 1713208646.676436 write 0x035994 thru 0x03b76c (0x5dd9 bytes) 000338 1713208646.677581 mapread 0x01da24 thru 0x020d15 (0x32f2 bytes) 000339 1713208646.680351 write 0x024d4f thru 0x02d5cf (0x8881 bytes) 000341 1713208646.682345 mapwrite 0x03b78e thru 0x03ffff (0x4872 bytes) 000342 1713208646.702903 write 0x01cd92 thru 0x01e12a (0x1399 bytes) 000343 1713208646.703622 mapwrite 0x03ec7e thru 0x03ffff (0x1382 bytes) 000345 1713208646.716570 mapread 0x03dba9 thru 0x03ffff (0x2457 bytes) 000347 1713208646.717285 mapread 0x03bb64 thru 0x03ffff (0x449c bytes) 000349 1713208646.718104 mapread 0x033446 thru 0x038de4 (0x599f bytes) 000351 1713208646.722257 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000352 1713208646.763803 trunc from 0x040000 to 0x01f3f4 (0x20c0b bytes) 000354 1713208646.767627 mapread 0x01c517 thru 0x01f3f4 (0x2ede bytes) 000357 1713208646.772086 mapread 0x0104cc thru 0x01276f (0x22a4 bytes) 000359 1713208646.775980 MIRROR_EXTEND to 4 mirrors 000360 1713208646.875078 trunc from 0x01f3f5 to 0x00e040 (0x113b4 bytes) 000363 1713208646.895635 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000365 1713208646.938845 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000366 1713208646.978634 mapread 0x00a0cd thru 0x00c9c7 (0x28fb bytes) 000369 1713208646.983774 write 0x01ec54 thru 0x02eb5c (0xff09 bytes) 000370 1713208646.984859 write 0x002155 thru 0x005ee5 (0x3d91 bytes) 000372 1713208646.987343 trunc from 0x02eb5d to 0x020fd8 (0xdb84 bytes) 000374 1713208646.991535 mapwrite 0x01caff thru 0x028632 (0xbb34 bytes) 000375 1713208647.015049 write 0x017eba thru 0x024b3d (0xcc84 bytes) 000376 1713208647.015832 mapwrite 0x016688 thru 0x019dbb (0x3734 bytes) 000377 1713208647.028471 mapwrite 0x01fd66 thru 0x02acc2 (0xaf5d bytes) 000379 1713208647.050375 read 0x007c9f thru 0x013f4f (0xc2b1 bytes) 000382 1713208647.052540 mapread 0x027eb1 thru 0x02acc2 (0x2e12 bytes) 000383 1713208647.053128 write 0x023ab8 thru 0x0300de (0xc627 bytes) 000385 1713208647.054216 mapwrite 0x01496b thru 0x01ab94 (0x622a bytes) 000387 1713208647.069530 mapread 0x01ee7f thru 0x02d6a5 (0xe827 bytes) 000389 1713208647.070258 mapread 0x01e430 thru 0x026283 (0x7e54 bytes) 000390 1713208647.070658 mapwrite 0x0317b5 thru 0x039b3f (0x838b bytes) 000391 1713208647.089845 write 0x006914 thru 0x012a9c (0xc189 bytes) 000392 1713208647.090542 trunc from 0x039b40 to 0x00440d (0x35732 bytes) 000393 1713208647.093978 mapwrite 0x003679 thru 0x0134e6 (0xfe6e bytes) 000394 1713208647.117483 trunc from 0x0134e7 to 0x03b38d (0x27ea7 bytes) 000395 1713208647.121417 mapwrite 0x030506 thru 0x036549 (0x6044 bytes) 000396 1713208647.138504 mapwrite 0x03f5fa thru 0x03ffff (0xa06 bytes) 000397 1713208647.158350 MIRROR_EXTEND to 3 mirrors 000405 1713208647.258193 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000406 1713208647.308211 mapwrite 0x003a87 thru 0x00f424 (0xb99e bytes) 000407 1713208647.354075 read 0x00f45d thru 0x00f98f (0x533 bytes) 000408 1713208647.354398 mapread 0x000818 thru 0x00dc5a (0xd443 bytes) 000409 1713208647.356018 trunc from 0x040000 to 0x01079d (0x2f862 bytes) 000410 1713208647.363719 mapread 0x00528e thru 0x01079d (0xb510 bytes) 000413 1713208647.364689 mapwrite 0x02b266 thru 0x030953 (0x56ee bytes) 000414 1713208647.384223 write 0x0016d5 thru 0x00b5c8 (0x9ef4 bytes) 000415 1713208647.384865 write 0x013ecc thru 0x0208d6 (0xca0b bytes) 000420 1713208647.389056 read 0x004947 thru 0x01216d (0xd827 bytes) 000422 1713208647.391105 MIRROR_RESYNC 2 mirrors 000424 1713208647.478116 read 0x016f2e thru 0x0264a9 (0xf57c bytes) 000425 1713208647.480760 read 0x011c5d thru 0x01b468 (0x980c bytes) 000427 1713208647.483808 mapwrite 0x002d7b thru 0x00e29a (0xb520 bytes) 000428 1713208647.536666 trunc from 0x030954 to 0x005339 (0x2b61a bytes) 000430 1713208647.545884 mapwrite 0x017cf3 thru 0x01fcb2 (0x7fc0 bytes) 000431 1713208647.575860 MIRROR_RESYNC 2 mirrors 000433 1713208647.652781 read 0x015837 thru 0x01fcb2 (0xa47c bytes) 000434 1713208647.654846 write 0x018f48 thru 0x0273b2 (0xe46b bytes) 000435 1713208647.668114 mapwrite 0x013f25 thru 0x0147da (0x8b6 bytes) 000437 1713208647.687125 trunc from 0x0273b3 to 0x014d6c (0x12646 bytes) 000439 1713208647.696539 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000442 1713208647.722889 read 0x00aaa8 thru 0x013046 (0x859f bytes) 000444 1713208647.725476 mapread 0x00fa4d thru 0x01123f (0x17f3 bytes) 000446 1713208647.726142 mapwrite 0x02700e thru 0x0302c8 (0x92bb bytes) 000448 1713208647.748884 trunc from 0x0302c9 to 0x039a92 (0x97ca bytes) 000449 1713208647.752275 read 0x00c022 thru 0x00d347 (0x1326 bytes) 000450 1713208647.752568 mapwrite 0x00e42b thru 0x015497 (0x706d bytes) 000452 1713208647.770754 MIRROR_EXTEND to 2 mirrors 000453 1713208647.871700 MIRROR_EXTEND to 3 mirrors 000454 1713208647.967250 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000455 1713208648.014746 read 0x0367c1 thru 0x039a92 (0x32d2 bytes) 000456 1713208648.016193 read 0x02337c thru 0x02b2db (0x7f60 bytes) 000460 1713208648.020585 MIRROR_EXTEND to 3 mirrors 000461 1713208648.128220 trunc from 0x039a93 to 0x01c05f (0x1da33 bytes) 000462 1713208648.143415 mapwrite 0x00c094 thru 0x011973 (0x58e0 bytes) 000464 1713208648.161486 read 0x00fe91 thru 0x010682 (0x7f2 bytes) 000465 1713208648.161769 mapread 0x002163 thru 0x002943 (0x7e1 bytes) 000466 1713208648.164145 read 0x01abf6 thru 0x01c05f (0x146a bytes) 000467 1713208648.167151 MIRROR_EXTEND to 4 mirrors 000468 1713208648.262545 mapwrite 0x00f991 thru 0x019b0a (0xa17a bytes) 000470 1713208648.311764 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000471 1713208648.352320 mapread 0x009b3f thru 0x00b137 (0x15f9 bytes) 000472 1713208648.357948 MIRROR_EXTEND to 4 mirrors 000473 1713208648.454619 read 0x0011e3 thru 0x00c191 (0xafaf bytes) 000476 1713208648.459384 trunc from 0x01c060 to 0x026c54 (0xabf5 bytes) 000477 1713208648.478379 MIRROR_RESYNC 4 mirrors 000479 1713208648.602183 mapwrite 0x025994 thru 0x02a137 (0x47a4 bytes) 000480 1713208648.637494 write 0x02134d thru 0x02b237 (0x9eeb bytes) 000482 1713208648.638330 trunc from 0x02b238 to 0x01616a (0x150cd bytes) 000483 1713208648.644029 MIRROR_RESYNC 4 mirrors 000484 1713208648.763841 read 0x0091a5 thru 0x01616a (0xcfc6 bytes) 000485 1713208648.766747 trunc from 0x01616b to 0x006a37 (0xf733 bytes) 000486 1713208648.785942 write 0x03820f thru 0x03f132 (0x6f24 bytes) 000488 1713208648.788190 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000489 1713208648.833781 MIRROR_RESYNC 3 mirrors 000492 1713208648.943945 mapread 0x02a16a thru 0x033ac3 (0x995a bytes) 000493 1713208648.948482 trunc from 0x03f133 to 0x01ab36 (0x245fc bytes) 000494 1713208648.965282 trunc from 0x01ab37 to 0x024570 (0x9a3a bytes) 000495 1713208648.969857 mapread 0x019b37 thru 0x024570 (0xaa3a bytes) 000497 1713208648.974563 write 0x0180d6 thru 0x0207c0 (0x86eb bytes) 000498 1713208648.975087 write 0x02fc4c thru 0x039100 (0x94b5 bytes) 000499 1713208648.976082 mapwrite 0x01a95c thru 0x026baa (0xc24f bytes) 000501 1713208648.994204 write 0x00efcb thru 0x012d05 (0x3d3b bytes) 000502 1713208648.996934 mapwrite 0x0340b5 thru 0x03f6b4 (0xb600 bytes) 000503 1713208649.016278 mapwrite 0x00de35 thru 0x017313 (0x94df bytes) 000504 1713208649.032543 mapwrite 0x026a59 thru 0x02b71c (0x4cc4 bytes) 000510 1713208649.047495 mapread 0x00875f thru 0x012980 (0xa222 bytes) 000511 1713208649.047964 trunc from 0x03f6b5 to 0x029eab (0x15809 bytes) 000512 1713208649.052467 mapread 0x025c66 thru 0x029917 (0x3cb2 bytes) 000514 1713208649.053393 mapwrite 0x03369d thru 0x03ffff (0xc963 bytes) 000517 1713208649.076181 write 0x00060a thru 0x003ca5 (0x369c bytes) 000519 1713208649.080775 MIRROR_RESYNC 3 mirrors 000520 1713208649.179242 mapread 0x00763d thru 0x00d3c8 (0x5d8c bytes) 000521 1713208649.183615 mapwrite 0x010b50 thru 0x020501 (0xf9b2 bytes) 000522 1713208649.226483 write 0x0325f5 thru 0x0362c4 (0x3cd0 bytes) 000523 1713208649.230211 MIRROR_EXTEND to 4 mirrors 000525 1713208649.339080 write 0x01b8de thru 0x01c0a6 (0x7c9 bytes) 000526 1713208649.353764 mapwrite 0x0397f0 thru 0x03ffff (0x6810 bytes) 000527 1713208649.369829 read 0x03bd0a thru 0x03ffff (0x42f6 bytes) 000528 1713208649.370172 mapwrite 0x034db3 thru 0x03c869 (0x7ab7 bytes) 000529 1713208649.384610 read 0x007daa thru 0x0160c6 (0xe31d bytes) 000530 1713208649.387716 mapread 0x030ff1 thru 0x03ffff (0xf00f bytes) 000533 1713208649.390375 MIRROR_RESYNC 4 mirrors 000536 1713208649.501765 read 0x02b524 thru 0x030f5c (0x5a39 bytes) 000540 1713208649.503799 read 0x002fe4 thru 0x00a9a0 (0x79bd bytes) 000541 1713208649.506480 mapwrite 0x00a408 thru 0x011133 (0x6d2c bytes) 000542 1713208649.553631 trunc from 0x040000 to 0x007776 (0x38889 bytes) 000543 1713208649.561635 write 0x00eccf thru 0x01871a (0x9a4c bytes) 000547 1713208649.563346 mapwrite 0x0224f2 thru 0x02e050 (0xbb5f bytes) 000549 1713208649.597909 mapread 0x01630a thru 0x0220bc (0xbdb3 bytes) 000550 1713208649.606911 mapwrite 0x03dcf7 thru 0x03ffff (0x2309 bytes) 000552 1713208649.628516 mapwrite 0x00dc2c thru 0x00f7c7 (0x1b9c bytes) 000554 1713208649.641782 write 0x011219 thru 0x01bd7e (0xab66 bytes) 000555 1713208649.642387 read 0x0086f3 thru 0x014953 (0xc261 bytes) 000557 1713208649.645310 mapwrite 0x0231ee thru 0x0280f6 (0x4f09 bytes) 000558 1713208649.657953 mapwrite 0x0247af thru 0x02acce (0x6520 bytes) 000561 1713208649.670388 mapwrite 0x034eca thru 0x036010 (0x1147 bytes) 000562 1713208649.682950 mapread 0x00c943 thru 0x011d2d (0x53eb bytes) 000563 1713208649.684749 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000564 1713208649.728733 mapwrite 0x015929 thru 0x015a6c (0x144 bytes) 000565 1713208649.743962 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000569 1713208649.785246 mapwrite 0x027a32 thru 0x033763 (0xbd32 bytes) 000571 1713208649.809023 write 0x0321f4 thru 0x033a3d (0x184a bytes) 000574 1713208649.810093 mapwrite 0x005f7a thru 0x011454 (0xb4db bytes) 000576 1713208649.826339 mapread 0x01981e thru 0x01d8a9 (0x408c bytes) 000578 1713208649.828732 write 0x03ace3 thru 0x03f723 (0x4a41 bytes) 000579 1713208649.830577 mapwrite 0x0173ff thru 0x025097 (0xdc99 bytes) 000580 1713208649.844260 read 0x00c8b5 thru 0x00cc97 (0x3e3 bytes) 000581 1713208649.844612 trunc from 0x040000 to 0x011935 (0x2e6ca bytes) 000582 1713208649.848983 read 0x007b17 thru 0x00ca7e (0x4f68 bytes) 000583 1713208649.849413 trunc from 0x011936 to 0x002c5a (0xecdb bytes) 000584 1713208649.853502 trunc from 0x002c5b to 0x039b3f (0x36ee5 bytes) 000588 1713208649.858433 trunc from 0x039b40 to 0x031acf (0x8070 bytes) 000589 1713208649.862887 trunc from 0x031ad0 to 0x01dd25 (0x13daa bytes) 000590 1713208649.866949 write 0x002aa6 thru 0x0047d9 (0x1d34 bytes) 000591 1713208649.869963 mapwrite 0x0135cb thru 0x01edf5 (0xb82b bytes) 000593 1713208649.890316 write 0x03e0fa thru 0x03ffff (0x1f06 bytes) 000594 1713208649.890980 read 0x0102cf thru 0x0165bd (0x62ef bytes) 000595 1713208649.891357 mapwrite 0x013ca2 thru 0x01e0c3 (0xa422 bytes) 000600 1713208649.908546 mapwrite 0x03c3b1 thru 0x03ffff (0x3c4f bytes) 000601 1713208649.922859 mapread 0x0262e5 thru 0x0300d4 (0x9df0 bytes) 000603 1713208649.923542 write 0x01139d thru 0x01fbf8 (0xe85c bytes) 000604 1713208649.924151 trunc from 0x040000 to 0x039bc1 (0x643e bytes) 000605 1713208649.929143 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000606 1713208649.958788 trunc from 0x039bc2 to 0x024245 (0x1597c bytes) 000608 1713208649.962520 read 0x001836 thru 0x00d909 (0xc0d4 bytes) 000610 1713208649.965160 trunc from 0x024246 to 0x01646e (0xddd7 bytes) 000612 1713208649.970103 MIRROR_EXTEND to 2 mirrors 000613 1713208650.091781 mapread 0x0071ae thru 0x01646e (0xf2c1 bytes) 000614 1713208650.100859 write 0x026a27 thru 0x032ea1 (0xc47b bytes) 000617 1713208650.116611 read 0x030d9c thru 0x032ea1 (0x2106 bytes) 000620 1713208650.117631 mapwrite 0x02b19c thru 0x032ab8 (0x791d bytes) 000622 1713208650.135921 trunc from 0x032ea2 to 0x01b68c (0x17815 bytes) 000623 1713208650.144114 MIRROR_EXTEND to 3 mirrors 000625 1713208650.268199 mapread 0x002fec thru 0x00996f (0x6984 bytes) 000629 1713208650.274246 MIRROR_RESYNC 3 mirrors 000631 1713208650.372069 mapwrite 0x01bc25 thru 0x01f1ec (0x35c8 bytes) 000632 1713208650.417276 mapwrite 0x002690 thru 0x00aa4e (0x83bf bytes) 000633 1713208650.439508 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000634 1713208650.497573 MIRROR_EXTEND to 3 mirrors 000635 1713208650.615035 mapwrite 0x023969 thru 0x02ea2b (0xb0c3 bytes) 000636 1713208650.653621 trunc from 0x02ea2c to 0x0050be (0x2996d bytes) 000637 1713208650.659631 read 0x0016ed thru 0x003635 (0x1f49 bytes) 000638 1713208650.661344 trunc from 0x0050bf to 0x039efc (0x34e3e bytes) 000639 1713208650.664991 read 0x038081 thru 0x039efc (0x1e7c bytes) 000640 1713208650.668844 mapread 0x003555 thru 0x010595 (0xd041 bytes) 000642 1713208650.669527 mapread 0x00a604 thru 0x0180a7 (0xdaa4 bytes) 000643 1713208650.669914 write 0x01e2e2 thru 0x02bf15 (0xdc34 bytes) 000644 1713208650.670595 read 0x00f95b thru 0x012a20 (0x30c6 bytes) 000647 1713208650.671248 mapwrite 0x0124a1 thru 0x01e603 (0xc163 bytes) 000648 1713208650.688800 write 0x013ce7 thru 0x0197c4 (0x5ade bytes) 000651 1713208650.690272 write 0x01c02a thru 0x023472 (0x7449 bytes) 000655 1713208650.694703 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000656 1713208650.742926 mapread 0x018682 thru 0x0200dc (0x7a5b bytes) 000657 1713208650.747592 mapread 0x00badb thru 0x0139a3 (0x7ec9 bytes) 000658 1713208650.752022 write 0x00c4f3 thru 0x014fbe (0x8acc bytes) 000659 1713208650.752908 read 0x01631c thru 0x01ac65 (0x494a bytes) 000660 1713208650.754674 MIRROR_RESYNC 2 mirrors 000662 1713208650.841224 write 0x029475 thru 0x02a1d6 (0xd62 bytes) 000663 1713208650.854687 trunc from 0x039efd to 0x039e19 (0xe3 bytes) 000665 1713208650.862596 mapwrite 0x006909 thru 0x00c9ba (0x60b2 bytes) 000666 1713208650.879209 write 0x023933 thru 0x02b4c8 (0x7b96 bytes) 000670 1713208650.882285 read 0x01078c thru 0x01455f (0x3dd4 bytes) 000671 1713208650.883776 read 0x033d74 thru 0x039e19 (0x60a6 bytes) 000676 1713208650.887098 trunc from 0x039e1a to 0x021545 (0x188d4 bytes) 000678 1713208650.891191 read 0x015b72 thru 0x021545 (0xb9d4 bytes) 000680 1713208650.891786 trunc from 0x021546 to 0x004e0a (0x1c73b bytes) 000681 1713208650.896487 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000682 1713208650.923263 mapwrite 0x023575 thru 0x02b5da (0x8066 bytes) 000683 1713208650.944847 write 0x018c67 thru 0x028762 (0xfafc bytes) 000686 1713208650.949325 MIRROR_EXTEND to 2 mirrors 000687 1713208651.062719 write 0x02de0c thru 0x035745 (0x793a bytes) 000688 1713208651.077816 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000694 1713208651.107599 trunc from 0x035746 to 0x03b5ff (0x5eba bytes) 000701 1713208651.113500 mapwrite 0x001eea thru 0x00240a (0x521 bytes) 000702 1713208651.130300 mapread 0x017ead thru 0x0203c4 (0x8518 bytes) 000703 1713208651.134168 mapwrite 0x00a7bc thru 0x00bf19 (0x175e bytes) 000706 1713208651.150126 write 0x02ccaf thru 0x03c652 (0xf9a4 bytes) 000707 1713208651.152997 write 0x00c2eb thru 0x01be10 (0xfb26 bytes) 000710 1713208651.154440 mapwrite 0x00bce7 thru 0x00fc94 (0x3fae bytes) 000712 1713208651.170705 mapwrite 0x027258 thru 0x034ef7 (0xdca0 bytes) 000714 1713208651.185044 trunc from 0x03c653 to 0x01e33a (0x1e318 bytes) 000715 1713208651.189205 mapwrite 0x01af8d thru 0x0271fe (0xc272 bytes) 000718 1713208651.210095 mapwrite 0x03059d thru 0x03f362 (0xedc6 bytes) 000720 1713208651.232489 write 0x00d2f2 thru 0x01c8e1 (0xf5f0 bytes) 000721 1713208651.233147 read 0x0030f6 thru 0x009c91 (0x6b9c bytes) 000725 1713208651.234421 mapread 0x0359e9 thru 0x03f362 (0x997a bytes) 000726 1713208651.234878 trunc from 0x03f363 to 0x00f6bb (0x2fca7 bytes) 000727 1713208651.238750 trunc from 0x00f6bc to 0x002225 (0xd496 bytes) 000729 1713208651.242644 mapread 0x000c79 thru 0x002225 (0x15ad bytes) 000730 1713208651.244217 write 0x02fe40 thru 0x0335d8 (0x3799 bytes) 000731 1713208651.244740 mapwrite 0x00b980 thru 0x01a2e6 (0xe967 bytes) 000732 1713208651.262590 trunc from 0x0335d9 to 0x0301ad (0x342b bytes) 000733 1713208651.266296 mapread 0x01e9e8 thru 0x0216d9 (0x2cf2 bytes) 000734 1713208651.268833 read 0x015d3c thru 0x019aec (0x3db1 bytes) 000736 1713208651.269284 trunc from 0x0301ae to 0x01e439 (0x11d74 bytes) 000737 1713208651.272751 read 0x000e3f thru 0x00d54f (0xc711 bytes) 000739 1713208651.274353 mapwrite 0x03a7d9 thru 0x03ffff (0x5827 bytes) 000740 1713208651.292324 mapwrite 0x0100fb thru 0x015698 (0x559e bytes) 000741 1713208651.304413 write 0x00d0fe thru 0x01bd08 (0xec0b bytes) 000742 1713208651.305338 read 0x00a238 thru 0x019b1c (0xf8e5 bytes) 000743 1713208651.306868 MIRROR_EXTEND to 2 mirrors 000745 1713208651.421475 write 0x0312d9 thru 0x03bc92 (0xa9ba bytes) 000746 1713208651.437230 trunc from 0x040000 to 0x036f50 (0x90af bytes) 000747 1713208651.445048 read 0x005054 thru 0x00ccaf (0x7c5c bytes) 000748 1713208651.447493 read 0x0322f6 thru 0x03435b (0x2066 bytes) 000749 1713208651.447855 mapwrite 0x0137d0 thru 0x014b8a (0x13bb bytes) 000750 1713208651.464604 mapwrite 0x000578 thru 0x002596 (0x201f bytes) 000752 1713208651.476128 read 0x00e539 thru 0x01780c (0x92d4 bytes) 000753 1713208651.477740 MIRROR_EXTEND to 3 mirrors 000754 1713208651.570492 mapwrite 0x007b37 thru 0x00d7e8 (0x5cb2 bytes) 000756 1713208651.612785 read 0x02e97c thru 0x034796 (0x5e1b bytes) 000760 1713208651.616146 read 0x0099b4 thru 0x018b68 (0xf1b5 bytes) 000762 1713208651.619601 read 0x021bb5 thru 0x02891c (0x6d68 bytes) 000763 1713208651.621362 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000764 1713208651.663945 read 0x0111a1 thru 0x016d1c (0x5b7c bytes) 000765 1713208651.666642 read 0x01cc66 thru 0x029eb2 (0xd24d bytes) 000766 1713208651.671034 MIRROR_RESYNC 2 mirrors 000768 1713208651.757984 MIRROR_EXTEND to 3 mirrors 000771 1713208651.855760 MIRROR_RESYNC 3 mirrors 000773 1713208651.888943 mapwrite 0x01415d thru 0x019bb3 (0x5a57 bytes) 000775 1713208651.930405 mapwrite 0x035cd3 thru 0x03ffff (0xa32d bytes) 000777 1713208651.954940 mapwrite 0x0026be thru 0x008700 (0x6043 bytes) 000779 1713208651.970375 read 0x036b6a thru 0x03e23f (0x76d6 bytes) 000780 1713208651.970703 mapwrite 0x033cbd thru 0x03c22f (0x8573 bytes) 000781 1713208651.983145 write 0x013b8a thru 0x01a139 (0x65b0 bytes) 000782 1713208651.983737 trunc from 0x040000 to 0x01c682 (0x2397d bytes) 000783 1713208651.987844 read 0x015ea3 thru 0x0195f4 (0x3752 bytes) 000786 1713208651.988632 trunc from 0x01c683 to 0x006b73 (0x15b0f bytes) 000789 1713208651.992773 write 0x02e5a6 thru 0x030410 (0x1e6b bytes) 000791 1713208651.993842 write 0x0160f7 thru 0x017fb1 (0x1ebb bytes) 000793 1713208651.998150 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000794 1713208652.040442 trunc from 0x030411 to 0x01c767 (0x13ca9 bytes) 000796 1713208652.043997 write 0x02f988 thru 0x03e35b (0xe9d4 bytes) 000797 1713208652.044798 mapread 0x014a44 thru 0x023418 (0xe9d5 bytes) 000798 1713208652.050508 mapread 0x02238a thru 0x02958a (0x7201 bytes) 000799 1713208652.050904 mapwrite 0x001ec8 thru 0x004ce5 (0x2e1e bytes) 000801 1713208652.066071 mapread 0x00a15f thru 0x00ee33 (0x4cd5 bytes) 000802 1713208652.067960 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000803 1713208652.094524 mapread 0x0023ae thru 0x00bbdb (0x982e bytes) 000804 1713208652.099656 write 0x022e20 thru 0x02f61c (0xc7fd bytes) 000805 1713208652.102821 mapwrite 0x024143 thru 0x02f9aa (0xb868 bytes) 000809 1713208652.117755 trunc from 0x03e35c to 0x01d450 (0x20f0b bytes) 000810 1713208652.121907 read 0x002794 thru 0x007d68 (0x55d5 bytes) 000811 1713208652.122418 trunc from 0x01d451 to 0x02ecbb (0x1186b bytes) 000812 1713208652.125640 mapread 0x01e0b2 thru 0x02bd4d (0xdc9c bytes) 000815 1713208652.130120 trunc from 0x02ecbc to 0x037841 (0x8b86 bytes) 000816 1713208652.133816 read 0x00d52a thru 0x017b7c (0xa653 bytes) 000817 1713208652.136140 trunc from 0x037842 to 0x0079f9 (0x2fe48 bytes) 000818 1713208652.139782 mapread 0x002efc thru 0x0079f9 (0x4afe bytes) 000820 1713208652.140492 write 0x039a7e thru 0x03ffff (0x6582 bytes) 000821 1713208652.141059 mapread 0x00521d thru 0x00ed15 (0x9af9 bytes) 000825 1713208652.150453 trunc from 0x040000 to 0x02806a (0x17f95 bytes) 000826 1713208652.154741 trunc from 0x02806b to 0x036719 (0xe6af bytes) 000828 1713208652.158860 mapread 0x0118f8 thru 0x01833b (0x6a44 bytes) 000829 1713208652.167947 trunc from 0x03671a to 0x024401 (0x12318 bytes) 000832 1713208652.172150 trunc from 0x024402 to 0x03e896 (0x1a495 bytes) 000833 1713208652.175692 read 0x015da4 thru 0x016569 (0x7c6 bytes) 000834 1713208652.175957 read 0x00167f thru 0x00cb39 (0xb4bb bytes) 000835 1713208652.176285 mapread 0x0387c6 thru 0x03e896 (0x60d1 bytes) 000837 1713208652.179634 write 0x004249 thru 0x0129c7 (0xe77f bytes) 000838 1713208652.180277 read 0x03d7b0 thru 0x03e896 (0x10e7 bytes) 000842 1713208652.181292 mapread 0x0080df thru 0x012e17 (0xad39 bytes) 000848 1713208652.182670 read 0x004903 thru 0x00ab81 (0x627f bytes) 000849 1713208652.182924 mapread 0x016768 thru 0x01fa4d (0x92e6 bytes) 000855 1713208652.184331 trunc from 0x03e897 to 0x038f9d (0x58f9 bytes) 000856 1713208652.188261 trunc from 0x038f9e to 0x0172da (0x21cc3 bytes) 000860 1713208652.192564 write 0x00ecb5 thru 0x016897 (0x7be3 bytes) 000862 1713208652.193287 trunc from 0x0172db to 0x02e273 (0x16f99 bytes) 000864 1713208652.197069 write 0x021789 thru 0x028a45 (0x72bd bytes) 000866 1713208652.200377 mapread 0x01f186 thru 0x02c2ba (0xd135 bytes) 000867 1713208652.201042 mapread 0x02d331 thru 0x02e273 (0xf43 bytes) 000868 1713208652.201661 mapread 0x010be7 thru 0x01a9b6 (0x9dd0 bytes) 000870 1713208652.202664 write 0x0327da thru 0x03952e (0x6d55 bytes) 000871 1713208652.203579 mapread 0x012069 thru 0x020fdb (0xef73 bytes) 000872 1713208652.204175 mapread 0x0185f1 thru 0x019536 (0xf46 bytes) 000873 1713208652.204623 mapwrite 0x01b17b thru 0x029b76 (0xe9fc bytes) 000874 1713208652.220128 mapwrite 0x01b57e thru 0x028f53 (0xd9d6 bytes) 000875 1713208652.235106 MIRROR_EXTEND to 2 mirrors 000878 1713208652.336613 read 0x01f277 thru 0x02dfd6 (0xed60 bytes) 000881 1713208652.339118 mapwrite 0x03cd2e thru 0x03ffff (0x32d2 bytes) 000882 1713208652.374744 mapread 0x03fafd thru 0x03ffff (0x503 bytes) 000886 1713208652.375855 write 0x019c74 thru 0x01ad7e (0x110b bytes) 000888 1713208652.378589 mapread 0x032dae thru 0x03362b (0x87e bytes) 000890 1713208652.380959 trunc from 0x040000 to 0x00444f (0x3bbb0 bytes) 000891 1713208652.385636 read 0x0007f3 thru 0x00444f (0x3c5d bytes) 000895 1713208652.388484 mapread 0x002fb7 thru 0x00444f (0x1499 bytes) 000898 1713208652.389433 mapread 0x004231 thru 0x00444f (0x21f bytes) 000899 1713208652.389862 read 0x000fe7 thru 0x00401f (0x3039 bytes) 000900 1713208652.391570 MIRROR_RESYNC 2 mirrors 000901 1713208652.470082 MIRROR_EXTEND to 3 mirrors 000903 1713208652.561529 MIRROR_RESYNC 3 mirrors 000910 1713208652.591682 mapread 0x00007a thru 0x00444f (0x43d6 bytes) 000911 1713208652.597325 MIRROR_RESYNC 3 mirrors 000912 1713208652.628799 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000913 1713208652.673807 write 0x0265cb thru 0x029c8f (0x36c5 bytes) 000914 1713208652.687550 trunc from 0x029c90 to 0x005926 (0x24369 bytes) 000916 1713208652.697931 MIRROR_EXTEND to 3 mirrors 000917 1713208652.789589 trunc from 0x005927 to 0x03df96 (0x38670 bytes) 000918 1713208652.807857 read 0x039f33 thru 0x03df96 (0x4064 bytes) 000919 1713208652.809988 trunc from 0x03df97 to 0x02393c (0x1a65a bytes) 000921 1713208652.814855 read 0x011130 thru 0x011b79 (0xa4a bytes) 000922 1713208652.817360 write 0x02683e thru 0x02999f (0x3162 bytes) 000923 1713208652.817900 trunc from 0x0299a0 to 0x0239b0 (0x5fef bytes) 000924 1713208652.821579 trunc from 0x0239b1 to 0x03835b (0x149ab bytes) 000926 1713208652.825289 read 0x0339fe thru 0x033b78 (0x17b bytes) 000927 1713208652.829219 MIRROR_EXTEND to 4 mirrors 000928 1713208652.928512 mapwrite 0x02f1d9 thru 0x034573 (0x539b bytes) 000929 1713208652.975009 trunc from 0x03835c to 0x00f2ac (0x290af bytes) 000930 1713208652.983526 read 0x00d66a thru 0x00f2ac (0x1c43 bytes) 000932 1713208652.985817 write 0x00509e thru 0x012343 (0xd2a6 bytes) 000934 1713208652.988425 mapwrite 0x036ee1 thru 0x03898f (0x1aaf bytes) 000937 1713208653.009888 mapread 0x00b972 thru 0x0114b6 (0x5b45 bytes) 000940 1713208653.010903 write 0x039fca thru 0x03e6b2 (0x46e9 bytes) 000943 1713208653.011891 write 0x00b6c4 thru 0x0138a2 (0x81df bytes) 000944 1713208653.013752 mapwrite 0x021040 thru 0x02a99c (0x995d bytes) 000945 1713208653.033729 MIRROR_EXTEND to 5 mirrors 000946 1713208653.140143 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000949 1713208653.181132 read 0x033a8e thru 0x03e6b2 (0xac25 bytes) 000950 1713208653.183605 write 0x035803 thru 0x03ffff (0xa7fd bytes) 000952 1713208653.187410 mapwrite 0x034671 thru 0x03ffff (0xb98f bytes) 000954 1713208653.203800 mapwrite 0x02141e thru 0x0300e5 (0xecc8 bytes) 000960 1713208653.223863 mapread 0x024804 thru 0x030f9e (0xc79b bytes) 000961 1713208653.224293 mapwrite 0x03109c thru 0x03f68e (0xe5f3 bytes) 000964 1713208653.240923 mapwrite 0x013945 thru 0x01a005 (0x66c1 bytes) 000965 1713208653.257259 trunc from 0x040000 to 0x03bd6a (0x4295 bytes) 000967 1713208653.261721 mapwrite 0x03b00d thru 0x03ffff (0x4ff3 bytes) 000969 1713208653.279753 read 0x03dc7b thru 0x03ffff (0x2385 bytes) 000973 1713208653.281362 write 0x005d7c thru 0x0088a9 (0x2b2e bytes) 000974 1713208653.283788 read 0x02950a thru 0x0391c4 (0xfcbb bytes) 000975 1713208653.284087 mapwrite 0x0299c7 thru 0x02bf6b (0x25a5 bytes) 000976 1713208653.295876 trunc from 0x040000 to 0x00459b (0x3ba64 bytes) 000977 1713208653.300014 mapread 0x0034f7 thru 0x00459b (0x10a5 bytes) 000978 1713208653.300544 mapwrite 0x03f5e2 thru 0x03ffff (0xa1e bytes) 000980 1713208653.319871 write 0x01ee5b thru 0x02218d (0x3333 bytes) 000981 1713208653.321671 MIRROR_EXTEND to 5 mirrors 000985 1713208653.446002 write 0x012b7a thru 0x020d05 (0xe18c bytes) 000987 1713208653.462693 mapwrite 0x0028b8 thru 0x00abad (0x82f6 bytes) 000988 1713208653.478925 write 0x03e9c6 thru 0x03ffff (0x163a bytes) 000989 1713208653.481680 mapread 0x016dd8 thru 0x018a95 (0x1cbe bytes) 000990 1713208653.482182 mapwrite 0x02db2e thru 0x030f37 (0x340a bytes) 000992 1713208653.496978 mapread 0x0116b0 thru 0x01760e (0x5f5f bytes) 000995 1713208653.499982 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000996 1713208653.540100 mapread 0x031f05 thru 0x0343ea (0x24e6 bytes) 000997 1713208653.544299 trunc from 0x040000 to 0x007df2 (0x3820d bytes) 001000 1713208653.548694 mapwrite 0x0258a6 thru 0x02ac2b (0x5386 bytes) All operations completed A-OK! PASS 70a (11s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200a: stress test ===================== 15:17:36 (1713208656) Starting client: oleg414-client.virtnet: -o user_xattr,flock oleg414-server@tcp:/lustre /mnt/lustre2 Starting client: oleg414-client.virtnet: -o user_xattr,flock oleg414-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5775328 .. Extending file size to 8392320 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8867296 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 5863872 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7718112 .. Extending file size to 8947968 .. Extending file size to 7818080 .. Extending file size to 8844416 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9110880 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 9287 9288 9289 9291 9293 done 192.168.204.114@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg414-client.virtnet /mnt/lustre2 (opts:) 192.168.204.114@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg414-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (63s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 15:18:41 (1713208721) Starting client: oleg414-client.virtnet: -o user_xattr,flock oleg414-server@tcp:/lustre /mnt/lustre2 Starting client: oleg414-client.virtnet: -o user_xattr,flock oleg414-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 1746400 .. Extending file size to 6837024 .. failed Extending file size to 5378304 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8400832 .. Extending file size to 7826848 .. Extending file size to 8431488 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7885152 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8067840 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8862560 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8911104 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8944864 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8915072 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 12924 12925 12927 12928 12931 192.168.204.114@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg414-client.virtnet /mnt/lustre2 (opts:) 192.168.204.114@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg414-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (64s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 15:19:46 (1713208786) Starting client: oleg414-client.virtnet: -o user_xattr,flock oleg414-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4224 2204288 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2210688 4608 2204032 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 40960 7497728 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4224 2204288 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 2210688 4608 2204032 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 40960 7497728 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0351371 s, 59.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0715261 s, 29.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.114@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg414-client.virtnet /mnt/lustre2 (opts:) PASS 200c (5s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 15:19:53 (1713208793) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0259149 s, 80.9 MB/s PASS 202 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 203: mirror file preserve mirror ID === 15:19:56 (1713208796) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa8:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa8:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa9:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0xa6:0x0] } PASS 203 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 15:19:59 (1713208799) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0202094 s, 104 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xd7d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xd80:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xd81:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 15:20:02 (1713208802) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0283997 s, 73.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xab:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xab:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa8:0x0] } PASS 204b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 15:20:06 (1713208806) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0258686 s, 81.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xd7e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xd82:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xd7e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xd82:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xd83:0x0] } PASS 204c (22s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 15:20:30 (1713208830) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg414-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0169126 s, 62.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xac:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xac:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xad:0x0] } PASS 204d (40s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 15:21:12 (1713208872) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg414-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xd84:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xd85:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0222985 s, 94.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xd84:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xd85:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xd7f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xd84:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xd85:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xd7f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (43s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 15:21:57 (1713208917) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg414-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool oleg414-server: Pool lustre.test_204f created Adding targets to pool oleg414-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg414-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0240856 s, 87.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg414-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg414-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg414-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (52s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 205: lfs mirror extend to set prefer and nocompr flags ========================================================== 15:22:50 (1713208970) PASS 205 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 15:22:53 (1713208973) oleg414-server: Pool lustre.test_206 created Adding targets to pool oleg414-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg414-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg414-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg414-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg414-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (13s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 15:23:08 (1713208988) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0171183 s, 61.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00803773 s, 130 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x3a5a798c Mirror 3: 0x3a5a798c PASS 207 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 15:23:12 (1713208992) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 15:23:14 (1713208994) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 15:23:16 (1713208996) Starting client: oleg414-client.virtnet: -o user_xattr,flock oleg414-server@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0155368 s, 0.3 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2080000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xbb0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xbb0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xbb0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1c40000 Read this (4 bytes): BBBB 192.168.204.114@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg414-client.virtnet /mnt/lustre2 (opts:) PASS 209a (14s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 15:23:31 (1713209011) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00262821 s, 3.1 MB/s the very first read read with warm cache PASS 209b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 210a: handle broken mirrored lovea ==== 15:23:34 (1713209014) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00939214 s, 112 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x73 lmm_fid: [0x200000405:0x73:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x73 lmm_fid: [0x200000405:0x73:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb4:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611251 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-04-15 15:23:35.000000000 -0400 Modify: 2024-04-15 15:23:35.000000000 -0400 Change: 2024-04-15 15:23:35.000000000 -0400 Birth: - PASS 210a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 15:23:38 (1713209018) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) debug_raw_pointers=0 debug_raw_pointers=0 == sanity-flr test complete, duration 777 sec ============ 15:23:39 (1713209019) === sanity-flr: start cleanup 15:23:40 (1713209020) === === sanity-flr: finish cleanup 15:23:40 (1713209020) ===