-----============= acceptance-small: sanity ============----- Wed Aug 26 11:49:32 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg316-server: /home/green/git/lustre-release/lustre/tests/except/sanity.ex - see CLIENT_VERSION > v2_15_63-134-gdacc4b6d38 (34683240 > 34553734) for LU-15963, go 312 - see CLIENT_VERSION > v2_15_63-239-gab4ede45b4 (34683240 > 34553839) for LU-17856, go 56wa 56xe - see CLIENT_VERSION >= v2_15_7-16-gab522d54ad (34683240 >= 34539280) for LU-16203, go 60i - see CLIENT_VERSION > v2_16_54-54-ga2e3a2f5a3 (34683240 > 34616886) for LU-14590, go 0d - see CLIENT_VERSION >= v2_17_50-196-g63ee4af3bb (34683240 >= 34681540) for LU-19620, go 60b - see CLIENT_VERSION >= v2_17_50-34-g089f438ba9 (34683240 >= 34681378) for LU-19692, go 77k - see CLIENT_VERSION >= v2_17_50-71-g2cbe29c282 (34683240 >= 34681415) for LU-19644, go 150ib - see CLIENT_VERSION >= v2_17_50-72-ge1f272796c (34683240 >= 34681416) for LU-19743, go 27z - see CLIENT_VERSION >= v2_17_52-99-g84242c4358 (34683240 >= 34681955) for LU-20010, go 56aa - see CLIENT_VERSION >= v2_17_55-19-g2868627130 (34683240 >= 34682643) for LU-20435, go 430a - need MDS1_VERSION <= 2.16.61-1-g89cf292a8c2 (34683240 <= 34618625) for LU-18938, skip 360 - see MDS1_VERSION <= CLIENT_VERSION (34683240 <= 34683240) for LU-18562, go 270a - need MDS1_VERSION < v2_14_55-100-g8a84c7f9c7 (34683240 < 34486116) for LU-14927, skip 0f - need OST1_VERSION < v2_17_50-410-gbd8e7b9555 (34683240 < 34681754) for LU-12550, skip 216 excepting tests: 42a 42c 42b 118c 118d 407 119i 216 817 411a 130b 130c 130d 130e 130f 130g skipping tests SLOW=no: 27m 60i 64b 68 71 135 136 230d 300o 842 51b 51c 51e 834 === sanity: start setup 11:50:01 (1787759401) === oleg316-client.virtnet: executing check_config_client /mnt/lustre oleg316-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg316-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff95a644c93800.idle_timeout=debug osc.lustre-OST0001-osc-ffff95a644c93800.idle_timeout=debug disable quota as required oleg316-server: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity: finish setup 11:50:36 (1787759436) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7636] preparing for tests involving mounts mke2fs 1.47.3-wc2 (11-Nov-2025) debug=all == sanity test 60a: llog_test run from kernel module and test llog_reader ========================================================== 11:50:43 (1787759443) SKIP: sanity test_60a missing subtest run-llog.sh SKIP 60a (6s) == sanity test 60b: limit repeated messages from CERROR/CWARN ========================================================== 11:50:49 (1787759449) PASS 60b (9s) == sanity test 60c: unlink file when mds full ============ 11:50:59 (1787759459) create 5000 files - open/close 438 (time 1787759473.95 total 10.02 last 43.73) - open/close 819 (time 1787759483.96 total 20.02 last 38.08) - open/close 1297 (time 1787759493.96 total 30.03 last 47.77) - open/close 1852 (time 1787759503.97 total 40.03 last 55.47) - open/close 2256 (time 1787759513.98 total 50.05 last 40.34) - open/close 2629 (time 1787759523.99 total 60.05 last 37.29) - open/close 3035 (time 1787759534.01 total 70.07 last 40.53) - open/close 3497 (time 1787759544.01 total 80.08 last 46.16) - open/close 4043 (time 1787759554.03 total 90.10 last 54.49) - open/close 4540 (time 1787759564.05 total 100.11 last 49.65) total: 5000 open/close in 109.23 seconds: 45.78 ops/second fail_loc=0x80000137 - unlinked 0 (time 1787759578 ; total 0 ; last 0) total: 5000 unlinks in 71 seconds: 70.422539 unlinks/second fail_loc=0 PASS 60c (201s) == sanity test 60d: test printk console message masking == 11:54:20 (1787759660) printk=0 emerg PASS 60d (9s) == sanity test 60e: no space while new llog is being created ========================================================== 11:54:30 (1787759670) fail_loc=0x15b PASS 60e (7s) == sanity test 60f: change debug_path works ============== 11:54:38 (1787759678) debug_path=/tmp/f60f.sanity fail_loc=0x8000050e ls: cannot access '/tmp/f60f.sanity*': No such file or directory 0 /tmp/f60f.sanity.1787759680.13912 debug_path=/tmp/lustre-log PASS 60f (8s) == sanity test 60g: transaction abort won't cause MDT hung ========================================================== 11:54:46 (1787759686) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5174: 14656 Killed ( local index=0; while true; do $LFS setdirstripe -i $(($index % $MDSCOUNT)) -c $MDSCOUNT $DIR/$tdir/subdir$index 2> /dev/null; mkdir $DIR/$tdir/subdir$index 2> /dev/null; rmdir $DIR/$tdir/subdir$index 2> /dev/null; index=$((index + 1)); done ) Started LFSCK on the device lustre-MDT0000: scrub namespace /mnt/lustre/d60g.sanity: subdir101 subdir108 subdir11 subdir141 subdir148 subdir162 subdir174 subdir181 subdir19 subdir213 subdir236 subdir244 subdir302 subdir309 subdir343 subdir350 subdir356 subdir370 subdir403 subdir42 subdir430 subdir437 subdir459 subdir466 subdir474 subdir480 subdir486 subdir499 subdir523 subdir530 subdir552 subdir56 subdir564 subdir578 subdir584 subdir590 subdir606 subdir612 subdir618 subdir626 subdir63 subdir632 subdir658 subdir665 subdir678 subdir693 subdir70 subdir707 subdir714 subdir728 subdir79 subdir87 /mnt/lustre/d60g.sanity/subdir101: /mnt/lustre/d60g.sanity/subdir108: /mnt/lustre/d60g.sanity/subdir11: /mnt/lustre/d60g.sanity/subdir141: /mnt/lustre/d60g.sanity/subdir148: /mnt/lustre/d60g.sanity/subdir162: /mnt/lustre/d60g.sanity/subdir174: /mnt/lustre/d60g.sanity/subdir181: /mnt/lustre/d60g.sanity/subdir19: /mnt/lustre/d60g.sanity/subdir213: /mnt/lustre/d60g.sanity/subdir236: /mnt/lustre/d60g.sanity/subdir244: /mnt/lustre/d60g.sanity/subdir302: /mnt/lustre/d60g.sanity/subdir309: /mnt/lustre/d60g.sanity/subdir343: /mnt/lustre/d60g.sanity/subdir350: /mnt/lustre/d60g.sanity/subdir356: /mnt/lustre/d60g.sanity/subdir370: /mnt/lustre/d60g.sanity/subdir403: /mnt/lustre/d60g.sanity/subdir42: /mnt/lustre/d60g.sanity/subdir430: /mnt/lustre/d60g.sanity/subdir437: /mnt/lustre/d60g.sanity/subdir459: /mnt/lustre/d60g.sanity/subdir466: /mnt/lustre/d60g.sanity/subdir474: /mnt/lustre/d60g.sanity/subdir480: /mnt/lustre/d60g.sanity/subdir486: /mnt/lustre/d60g.sanity/subdir499: /mnt/lustre/d60g.sanity/subdir523: /mnt/lustre/d60g.sanity/subdir530: /mnt/lustre/d60g.sanity/subdir552: /mnt/lustre/d60g.sanity/subdir56: /mnt/lustre/d60g.sanity/subdir564: /mnt/lustre/d60g.sanity/subdir578: /mnt/lustre/d60g.sanity/subdir584: /mnt/lustre/d60g.sanity/subdir590: /mnt/lustre/d60g.sanity/subdir606: /mnt/lustre/d60g.sanity/subdir612: /mnt/lustre/d60g.sanity/subdir618: /mnt/lustre/d60g.sanity/subdir626: /mnt/lustre/d60g.sanity/subdir63: /mnt/lustre/d60g.sanity/subdir632: /mnt/lustre/d60g.sanity/subdir658: /mnt/lustre/d60g.sanity/subdir665: /mnt/lustre/d60g.sanity/subdir678: /mnt/lustre/d60g.sanity/subdir693: /mnt/lustre/d60g.sanity/subdir70: /mnt/lustre/d60g.sanity/subdir707: /mnt/lustre/d60g.sanity/subdir714: /mnt/lustre/d60g.sanity/subdir728: /mnt/lustre/d60g.sanity/subdir79: /mnt/lustre/d60g.sanity/subdir87: PASS 60g (132s) == sanity test 60h: striped directory with missing stripes can be accessed ========================================================== 11:56:58 (1787759818) SKIP: sanity test_60h Need at least 2 MDTs SKIP 60h (4s) SKIP: sanity test_60i skipping SLOW test 60i == sanity test 60j: llog_reader reports corruptions ====== 11:57:05 (1787759825) SKIP: sanity test_60j ldiskfs only test SKIP 60j (5s) == sanity test 61a: mmap() writes don't make sync hang ========================================================================== 11:57:09 (1787759829) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0331873 s, 123 kB/s PASS 61a (8s) == sanity test 61b: mmap() of unstriped file is successful ========================================================== 11:57:17 (1787759837) PASS 61b (8s) == sanity test 63a: Verify oig_wait interruption does not crash ================================================================= 11:57:26 (1787759846) /home/green/git/lustre-release/lustre/tests/sanity.sh: line 11075: 20273 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 11075: 20282 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 11075: 20287 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 11075: 20291 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 11075: 20295 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 11075: 20298 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 11075: 20302 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 11075: 20306 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 11075: 20310 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 11075: 20314 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k checking grant......UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2432 2206208 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 15360 3751936 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 17408 7519232 1% /mnt/lustre pass grant check: client:177209344 server:177209344 PASS 63a (71s) == sanity test 63b: async write errors should be returned to fsync ============================================================= 11:58:37 (1787759917) debug=-1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0123195 s, 332 kB/s fail_loc=0x80000406 fsync: Input/output error checking grant......UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2432 2206208 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3765248 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7532544 1% /mnt/lustre pass grant check: client:177209344 server:177209344 PASS 63b (14s) == sanity test 63c: test sync_on_close=1 ================= 11:58:51 (1787759931) llite.lustre-ffff95a644c93800.sync_on_close=1 total: 20 open/close in 2.35 seconds: 8.50 ops/second total 202 -rw-r--r-- 1 root root 90112 Aug 26 11:59 f63c.sanity-1 -rw-r--r-- 1 root root 6 Aug 26 11:58 f63c.sanity-empty-0 -rw-r--r-- 1 root root 6 Aug 26 11:58 f63c.sanity-empty-1 -rw-r--r-- 1 root root 7 Aug 26 11:58 f63c.sanity-empty-10 -rw-r--r-- 1 root root 7 Aug 26 11:58 f63c.sanity-empty-11 -rw-r--r-- 1 root root 7 Aug 26 11:58 f63c.sanity-empty-12 -rw-r--r-- 1 root root 7 Aug 26 11:58 f63c.sanity-empty-13 -rw-r--r-- 1 root root 7 Aug 26 11:58 f63c.sanity-empty-14 -rw-r--r-- 1 root root 7 Aug 26 11:58 f63c.sanity-empty-15 -rw-r--r-- 1 root root 7 Aug 26 11:58 f63c.sanity-empty-16 -rw-r--r-- 1 root root 7 Aug 26 11:58 f63c.sanity-empty-17 -rw-r--r-- 1 root root 7 Aug 26 11:58 f63c.sanity-empty-18 -rw-r--r-- 1 root root 7 Aug 26 11:58 f63c.sanity-empty-19 -rw-r--r-- 1 root root 6 Aug 26 11:58 f63c.sanity-empty-2 -rw-r--r-- 1 root root 6 Aug 26 11:58 f63c.sanity-empty-3 -rw-r--r-- 1 root root 6 Aug 26 11:58 f63c.sanity-empty-4 -rw-r--r-- 1 root root 6 Aug 26 11:58 f63c.sanity-empty-5 -rw-r--r-- 1 root root 6 Aug 26 11:58 f63c.sanity-empty-6 -rw-r--r-- 1 root root 6 Aug 26 11:58 f63c.sanity-empty-7 -rw-r--r-- 1 root root 6 Aug 26 11:58 f63c.sanity-empty-8 -rw-r--r-- 1 root root 6 Aug 26 11:58 f63c.sanity-empty-9 error: set_param: setting /sys/kernel/debug/lnet/debug=rpctrace+dlmtrace+inode+entry: Invalid argument error: set_param: setting 'debug'='rpctrace+dlmtrace+inode+entry': Invalid argument 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 3.60349 s, 4.7 MB/s llite.lustre-ffff95a644c93800.sync_on_close=0 PASS 63c (29s) == sanity test 64a: verify filter grant calculations (in kernel) =============================================================== 11:59:20 (1787759960) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2560 2206080 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 10240 3759104 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 10240 3759104 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 20480 7518208 1% /mnt/lustre osc.lustre-OST0000-osc-ffff95a644c93800.cur_grant_bytes=195952640 osc.lustre-OST0001-osc-ffff95a644c93800.cur_grant_bytes=105644032 checking grant......UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2560 2206080 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 10240 3759104 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 10240 3759104 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 20480 7518208 1% /mnt/lustre pass grant check: client:301596672 server:301596672 PASS 64a (12s) SKIP: sanity test_64b skipping SLOW test 64b == sanity test 64c: verify grant shrink ================== 11:59:34 (1787759974) osc.lustre-OST0000-osc-ffff95a644c93800.cur_grant_bytes=0 checking grant......UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2560 2206080 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 10240 3759104 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 10240 3759104 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 20480 7518208 1% /mnt/lustre pass grant check: client:301596672 server:301596672 PASS 64c (11s) == sanity test 64d: check grant limit exceed ============= 11:59:44 (1787759984) 1000+0 records in 1000+0 records out 1048576000 bytes (1.0 GB, 1000 MiB) copied, 35.4674 s, 29.6 MB/s /home/green/git/lustre-release/lustre/tests/sanity.sh: line 11329: kill: (23875) - No such process checking grant......UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2560 2206080 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 1007616 2717696 28% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 10240 3759104 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 1017856 6476800 14% /mnt/lustre pass grant check: client:590741504 server:590741504 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 64d (68s) == sanity test 64e: check grant consumption (no grant allocation) ========================================================== 12:00:52 (1787760052) debug=+cache Stopping client oleg316-client.virtnet /mnt/lustre (opts:) Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre fail_loc=0x725 1+0 records in 1+0 records out 622592 bytes (623 kB, 608 KiB) copied, 0.190444 s, 3.3 MB/s fail_loc=0 Stopping client oleg316-client.virtnet /mnt/lustre (opts:) Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre fail_loc=0x725 fail_loc=0 PASS 64e (25s) == sanity test 64f: check grant consumption (with grant allocation) ========================================================== 12:01:17 (1787760077) debug=+cache Stopping client oleg316-client.virtnet /mnt/lustre (opts:) Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre 1+0 records in 1+0 records out 651264 bytes (651 kB, 636 KiB) copied, 0.368818 s, 1.8 MB/s Stopping client oleg316-client.virtnet /mnt/lustre (opts:) Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre llite.lustre-ffff95a645874800.hybrid_io=0 fail_loc=0x50a fail_val=3 1+0 records in 1+0 records out 651264 bytes (651 kB, 636 KiB) copied, 0.11654 s, 5.6 MB/s fail_loc=0 fail_val=0 PASS 64f (18s) == sanity test 64g: grant shrink on MDT ================== 12:01:35 (1787760095) 1+0 records in 1+0 records out 131072 bytes (131 kB, 128 KiB) copied, 0.0162463 s, 8.1 MB/s 1+0 records in 1+0 records out 131072 bytes (131 kB, 128 KiB) copied, 0.0151638 s, 8.6 MB/s 1+0 records in 1+0 records out 131072 bytes (131 kB, 128 KiB) copied, 0.0199068 s, 6.6 MB/s 1+0 records in 1+0 records out 131072 bytes (131 kB, 128 KiB) copied, 0.0169214 s, 7.7 MB/s 1+0 records in 1+0 records out 131072 bytes (131 kB, 128 KiB) copied, 0.0557876 s, 2.3 MB/s 1+0 records in 1+0 records out 131072 bytes (131 kB, 128 KiB) copied, 0.013717 s, 9.6 MB/s 1+0 records in 1+0 records out 131072 bytes (131 kB, 128 KiB) copied, 0.0243729 s, 5.4 MB/s 1+0 records in 1+0 records out 131072 bytes (131 kB, 128 KiB) copied, 0.0190858 s, 6.9 MB/s 1+0 records in 1+0 records out 131072 bytes (131 kB, 128 KiB) copied, 0.0220918 s, 5.9 MB/s 1+0 records in 1+0 records out 131072 bytes (131 kB, 128 KiB) copied, 0.027712 s, 4.7 MB/s 3932160 grants, 0 pages 3932160 grants, 8 pages 3932160 grants, 8 pages 3932160 grants, 0 pages mdc.lustre-MDT0000-mdc-ffff95a645874800.grant_shrink_interval=5 0 grants, 8 pages 0 grants, 8 pages 0 grants, 0 pages mdc.lustre-MDT0000-mdc-ffff95a645874800.grant_shrink_interval=1200 PASS 64g (27s) == sanity test 64h: grant shrink on read ================= 12:02:03 (1787760123) osc.lustre-OST0000-osc-ffff95a645874800.grant_shrink=1 osc.lustre-OST0000-osc-ffff95a645874800.grant_shrink_interval=10 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 2.89012 s, 3.6 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0420813 s, 97.3 kB/s PASS 64h (25s) == sanity test 64i: shrink on reconnect ================== 12:02:28 (1787760148) 64+0 records in 64+0 records out 67108864 bytes (67 MB, 64 MiB) copied, 8.51799 s, 7.9 MB/s fail_loc=0x80000513 fail_val=17 osc.lustre-OST0000-osc-ffff95a645874800.cur_grant_bytes=64749568B Failing ost1 on oleg316-server Stopping /mnt/lustre-ost1 (opts:) on oleg316-server 12:02:45 (1787760165) shut down facet: ost1 facet_host: oleg316-server facet_failover_host: oleg316-server Failover ost1 to oleg316-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg316-server: oleg316-server.virtnet: executing set_default_debug all all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-OST0000 12:03:14 (1787760194) targets are mounted 12:03:14 (1787760194) facet_failover done oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 1.54972 s, 5.4 MB/s PASS 64i (68s) == sanity test 64j: check grants on re-done rpc ========== 12:03:37 (1787760217) fail_loc=0x80000256 checking grant......UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2560 2206080 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 10240 3757056 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 10240 3759104 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 20480 7516160 1% /mnt/lustre pass grant check: client:22544384 server:22544384 PASS 64j (12s) == sanity test 65a: directory with no stripe info ======== 12:03:50 (1787760230) default stripe 1, ost count 2 PASS 65a (9s) == sanity test 65b: directory setstripe -S stripe_size*2 -i 0 -c 1 ========================================================== 12:03:59 (1787760239) dir stripe 1, default stripe 1, ost count 2 PASS 65b (8s) == sanity test 65c: directory setstripe -S stripe_size*4 -i 1 -c 1 ========================================================== 12:04:07 (1787760247) dir stripe 1, default stripe 1, ost count 2 PASS 65c (7s) == sanity test 65d: directory setstripe -S stripe_size -c stripe_count ========================================================== 12:04:14 (1787760254) dir stripe 0, default stripe 1, ost count 2 PASS 65d (7s) == sanity test 65e: directory setstripe defaults ========= 12:04:21 (1787760261) (Default) /mnt/lustre/d65e.sanity default stripe 1, ost count 2 PASS 65e (7s) == sanity test 65f: dir setstripe permission (should return error) ============================================================= 12:04:29 (1787760269) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [setstripe] [/mnt/lustre/d65f.sanityf] lfs setstripe: setstripe error for '/mnt/lustre/d65f.sanityf': Operation not permitted PASS 65f (7s) == sanity test 65g: directory setstripe -d =============== 12:04:36 (1787760276) (Default) /mnt/lustre/d65g.sanity PASS 65g (7s) == sanity test 65h: directory stripe info inherit ============================================================================== 12:04:44 (1787760284) PASS 65h (7s) == sanity test 65i: various tests to set root directory striping ========================================================== 12:04:50 (1787760290) /mnt/lustre stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/d65f.sanityf stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/f60b.sanity lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2 0x2 0x280000400 /mnt/lustre/f61 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2504 0x9c8 0x280000400 /mnt/lustre/d65d.sanity stripe_count: 1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 /mnt/lustre/d63c.sanity stripe_count: 1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 /mnt/lustre/f61b.sanity has no stripe info /mnt/lustre/d65b.sanity stripe_count: 1 stripe_size: 8388608 pattern: raid0 stripe_offset: 0 /mnt/lustre/d60f.sanity stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/d65c.sanity stripe_count: 1 stripe_size: 16777216 pattern: raid0 stripe_offset: 1 /mnt/lustre/f63b.sanity lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2505 0x9c9 0x240000400 /mnt/lustre/d65a.sanity stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/f64j.sanity lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2524 0x9dc 0x240000400 /mnt/lustre/d65g.sanity stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/d65e.sanity stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/d65h.sanity stripe_count: 1 stripe_size: 8388608 pattern: raid0 stripe_offset: 0 /mnt/lustre lmm_fid: [0x200000007:0x1:0x0] stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/d65f.sanityf lmm_fid: [0x200000406:0x29:0x0] stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/f60b.sanity lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000401 lmm_object_id: 0x3 lmm_fid: [0x200000401:0x3:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2 0x2 0x280000400 /mnt/lustre/f61 lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000401 lmm_object_id: 0x1943 lmm_fid: [0x200000401:0x1943:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2504 0x9c8 0x280000400 /mnt/lustre/d65d.sanity lmm_fid: [0x200000406:0x23:0x0] stripe_count: 1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 /mnt/lustre/d63c.sanity lmm_fid: [0x200000401:0x1950:0x0] stripe_count: 1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 /mnt/lustre/f61b.sanity has no stripe info /mnt/lustre/d65b.sanity lmm_fid: [0x200000406:0x1f:0x0] stripe_count: 1 stripe_size: 8388608 pattern: raid0 stripe_offset: 0 /mnt/lustre/d60f.sanity lmm_fid: [0x200000401:0x138d:0x0] stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/d65c.sanity lmm_fid: [0x200000406:0x21:0x0] stripe_count: 1 stripe_size: 16777216 pattern: raid0 stripe_offset: 1 /mnt/lustre/f63b.sanity lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000401 lmm_object_id: 0x194f lmm_fid: [0x200000401:0x194f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2505 0x9c9 0x240000400 /mnt/lustre/d65a.sanity lmm_fid: [0x200000406:0x1d:0x0] stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/f64j.sanity lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000406 lmm_object_id: 0x1c lmm_fid: [0x200000406:0x1c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2524 0x9dc 0x240000400 /mnt/lustre/d65g.sanity lmm_fid: [0x200000406:0x2a:0x0] stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/d65e.sanity lmm_fid: [0x200000406:0x26:0x0] stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/d65h.sanity lmm_fid: [0x200000406:0x2c:0x0] stripe_count: 1 stripe_size: 8388608 pattern: raid0 stripe_offset: 0 PASS 65i (8s) == sanity test 65j: set default striping on root directory (bug 6367)=========================================================== 12:04:59 (1787760299) PASS 65j (9s) == sanity test 65k: validate manual striping works properly with deactivated OSCs ========================================================== 12:05:08 (1787760308) Check OST status: lustre-OST0000-osc-MDT0000 is active lustre-OST0001-osc-MDT0000 is active - open/close 772 (time 1787760326.28 total 10.01 last 77.10) total: 1000 open/close in 13.14 seconds: 76.11 ops/second Deactivate: lustre-OST0000-osc-MDT0000 /home/green/git/lustre-release/lustre/utils/lfs setstripe -i 0 -c 1 /mnt/lustre/d65k.sanity/0 /home/green/git/lustre-release/lustre/utils/lfs setstripe -i 1 -c 1 /mnt/lustre/d65k.sanity/1 - unlinked 0 (time 1787760337 ; total 0 ; last 0) total: 1000 unlinks in 8 seconds: 125.000000 unlinks/second lustre-OST0000-osc-MDT0000 is Activate oleg316-server: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg316-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec - open/close 724 (time 1787760368.32 total 10.01 last 72.31) total: 1000 open/close in 14.51 seconds: 68.90 ops/second Deactivate: lustre-OST0001-osc-MDT0000 /home/green/git/lustre-release/lustre/utils/lfs setstripe -i 0 -c 1 /mnt/lustre/d65k.sanity/0 /home/green/git/lustre-release/lustre/utils/lfs setstripe -i 1 -c 1 /mnt/lustre/d65k.sanity/1 - unlinked 0 (time 1787760382 ; total 0 ; last 0) total: 1000 unlinks in 7 seconds: 142.857147 unlinks/second lustre-OST0001-osc-MDT0000 is Activate oleg316-server: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg316-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec PASS 65k (98s) == sanity test 65l: lfs find on -1 stripe dir ================================================================================== 12:06:46 (1787760406) PASS 65l (9s) == sanity test 65m: normal user can't set filesystem default stripe ========================================================== 12:06:55 (1787760415) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [setstripe] [-c] [2] [/mnt/lustre] lfs setstripe: setstripe error for '/mnt/lustre': Operation not permitted PASS 65m (10s) == sanity test 65n: don't inherit default layout from root for new subdirectories ========================================================== 12:07:06 (1787760426) Creating new pool test_65n oleg316-server: Pool lustre.test_65n created Adding targets to pool oleg316-server: OST lustre-OST0000_UUID added to pool lustre.test_65n oleg316-server: OST lustre-OST0001_UUID added to pool lustre.test_65n /home/green/git/lustre-release/lustre/utils/lfs getstripe -d /mnt/lustre/d65n.sanity-4 stripe_count: 1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 pool: test_65n /home/green/git/lustre-release/lustre/utils/lfs getstripe -d /mnt/lustre stripe_count: 1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 pool: test_65n SKIP: sanity test_65n needs >= 2 MDTs Destroy the created pools: test_65n lustre.test_65n oleg316-server: OST lustre-OST0000_UUID removed from pool lustre.test_65n oleg316-server: OST lustre-OST0001_UUID removed from pool lustre.test_65n oleg316-server: Pool lustre.test_65n destroyed SKIP 65n (27s) == sanity test 65o: pool inheritance for mdt component === 12:07:34 (1787760454) Creating new pool test_65o oleg316-server: Pool lustre.test_65o created Adding targets to pool oleg316-server: OST lustre-OST0000_UUID added to pool lustre.test_65o oleg316-server: OST lustre-OST0001_UUID added to pool lustre.test_65o /mnt/lustre/d65o.sanity lcm_layout_gen: 0 lcm_mirror_count: 1 lcm_entry_count: 2 lcme_id: N/A lcme_mirror_id: N/A lcme_flags: 0 lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 stripe_count: 0 stripe_size: 1048576 pattern: mdt mdt_index: -1 lcme_id: N/A lcme_mirror_id: N/A lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF stripe_count: 1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 pool: test_65o /mnt/lustre/d65o.sanity/dir2 lcm_layout_gen: 0 lcm_mirror_count: 1 lcm_entry_count: 2 lcme_id: N/A lcme_mirror_id: N/A lcme_flags: 0 lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 stripe_count: 0 stripe_size: 1048576 pattern: mdt mdt_index: -1 lcme_id: N/A lcme_mirror_id: N/A lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF stripe_count: 1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 pool: test_65o lcm_layout_gen: 0 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: N/A lcme_mirror_id: N/A lcme_flags: 0 lcme_extent.e_start: 0 lcme_extent.e_end: EOF stripe_count: 1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 pool: test_65o Destroy the created pools: test_65o lustre.test_65o oleg316-server: OST lustre-OST0000_UUID removed from pool lustre.test_65o oleg316-server: OST lustre-OST0001_UUID removed from pool lustre.test_65o oleg316-server: Pool lustre.test_65o destroyed PASS 65o (31s) == sanity test 65p: setstripe with yaml file and huge number ========================================================== 12:08:04 (1787760484) PASS 65p (9s) == sanity test 65q: setstripe with >=8E offset should fail ========================================================== 12:08:14 (1787760494) Cannot set layout EA: Invalid argument lfs setstripe: lfs setstripe: cannot set default composite layout for '/mnt/lustre/d65q.sanity/src_dir': Invalid argument (22) PASS 65q (11s) == sanity test 65r: prevent all-zero offsets ============= 12:08:25 (1787760505) PASS 65r (11s) == sanity test 66: update inode blocks count on client ========================================================================= 12:08:36 (1787760516) 8+0 records in 8+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.017455 s, 469 kB/s PASS 66 (17s) == sanity test 69: verify oa2dentry return -ENOENT doesn't LBUG ================================================================ 12:08:53 (1787760533) directio on /mnt/lustre/f69.sanity.2 for 1x4194304 bytes PASS fail_loc=0x217 directio on /mnt/lustre/f69.sanity for 2x4194304 bytes Write error No such file or directory (rc = -1, len = 8388608) fail_loc=0 directio on /mnt/lustre/f69.sanity for 2x4194304 bytes PASS directio on /mnt/lustre/f69.sanity for 1x4194304 bytes PASS fail_loc=0x217 directio on /mnt/lustre/f69.sanity for 1x4194304 bytes Read error: No such file or directory rc = -1 fail_loc=0 PASS 69 (15s) == sanity test 70a: verify health_check, health_write don't explode (on OST) ========================================================== 12:09:08 (1787760548) enable_health_write=off enable_health_write=0 enable_health_write=on enable_health_write=1 enable_health_write=0 PASS 70a (19s) SKIP: sanity test_71 skipping SLOW test 71 == sanity test 72a: Test that remove suid works properly (bug5695) ============================================================== 12:09:29 (1787760569) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7636] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/f72a.sanity] [bs=512] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.00634151 s, 80.7 kB/s PASS 72a (8s) == sanity test 72b: Test that we keep mode setting if without file data changed (bug 24226) ========================================================== 12:09:38 (1787760578) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7636] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/f72b.sanity-fg] chmod: changing permissions of '/mnt/lustre/f72b.sanity-fg': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/f72b.sanity-fu] chmod: changing permissions of '/mnt/lustre/f72b.sanity-fu': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/f72b.sanity-dg] chmod: changing permissions of '/mnt/lustre/f72b.sanity-dg': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/f72b.sanity-du] chmod: changing permissions of '/mnt/lustre/f72b.sanity-du': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [2777] [/mnt/lustre/f72b.sanity-fg] chmod: changing permissions of '/mnt/lustre/f72b.sanity-fg': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [2777] [/mnt/lustre/f72b.sanity-fu] chmod: changing permissions of '/mnt/lustre/f72b.sanity-fu': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [2777] [/mnt/lustre/f72b.sanity-dg] chmod: changing permissions of '/mnt/lustre/f72b.sanity-dg': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [2777] [/mnt/lustre/f72b.sanity-du] chmod: changing permissions of '/mnt/lustre/f72b.sanity-du': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [4777] [/mnt/lustre/f72b.sanity-fg] chmod: changing permissions of '/mnt/lustre/f72b.sanity-fg': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [4777] [/mnt/lustre/f72b.sanity-fu] chmod: changing permissions of '/mnt/lustre/f72b.sanity-fu': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [4777] [/mnt/lustre/f72b.sanity-dg] chmod: changing permissions of '/mnt/lustre/f72b.sanity-dg': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [4777] [/mnt/lustre/f72b.sanity-du] chmod: changing permissions of '/mnt/lustre/f72b.sanity-du': Operation not permitted PASS 72b (8s) == sanity test 73: multiple MDC requests (should not deadlock) ========================================================== 12:09:46 (1787760586) multiop /mnt/lustre/d73-1/f73-1 vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7636 fail_loc=0x80000129 fail_loc=0 /mnt/lustre/d73-1/f73-1 has type file OK /mnt/lustre/d73-1/f73-2 has type file OK /mnt/lustre/d73-2/f73-3 has type file OK PASS 73 (36s) == sanity test 74a: ldlm_enqueue freed-export error path, ls (shouldn't LBUG) ========================================================== 12:10:22 (1787760622) fail_loc=0x8000030e /mnt/lustre/f74a fail_loc=0 PASS 74a (9s) == sanity test 74b: ldlm_enqueue freed-export error path, touch (shouldn't LBUG) ========================================================== 12:10:31 (1787760631) fail_loc=0x8000030e fail_loc=0 PASS 74b (9s) == sanity test 74c: ldlm_lock_create error path, (shouldn't LBUG) ========================================================== 12:10:40 (1787760640) fail_loc=0x319 touch: cannot touch '/mnt/lustre/f74c.sanity': No such file or directory fail_loc=0 PASS 74c (7s) == sanity test 76a: confirm clients recycle inodes properly ============================================================== 12:10:47 (1787760647) before slab objects: 46 created: 512, after slab objects: 46 PASS 76a (85s) == sanity test 76b: confirm clients recycle directory inodes properly ============================================================== 12:12:12 (1787760732) slab objects before: 46, after: 46 PASS 76b (48s) == sanity test 77a: normal checksum read/write operation ========================================================== 12:13:00 (1787760780) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.294273 s, 28.5 MB/s 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 1.30745 s, 6.4 MB/s PASS 77a (9s) == sanity test 77b: checksum error on client write, read ========================================================== 12:13:09 (1787760789) fail_loc=0x80000409 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 1.0766 s, 7.8 MB/s fail_loc=0 set checksum type to crc32, rc = 0 fail_loc=0x80000408 fail_loc=0 set checksum type to adler, rc = 0 fail_loc=0x80000408 fail_loc=0 set checksum type to crc32c, rc = 0 fail_loc=0x80000408 fail_loc=0 set checksum type to t10ip512, rc = 0 fail_loc=0x80000408 fail_loc=0 set checksum type to t10ip4K, rc = 0 fail_loc=0x80000408 fail_loc=0 set checksum type to t10crc512, rc = 0 fail_loc=0x80000408 fail_loc=0 set checksum type to t10crc4K, rc = 0 fail_loc=0x80000408 fail_loc=0 set checksum type to crc32c, rc = 0 PASS 77b (41s) == sanity test 77c: checksum error on client read with debug ========================================================== 12:13:50 (1787760830) debug_path=/tmp/lustre-log 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 1.35668 s, 6.2 MB/s osc.lustre-OST0000-osc-ffff95a645874800.checksum_dump=1 osc.lustre-OST0001-osc-ffff95a645874800.checksum_dump=1 obdfilter.lustre-OST0000.checksum_dump=1 obdfilter.lustre-OST0001.checksum_dump=1 fail_loc=0x80000408 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 6.1741 s, 1.4 MB/s fail_loc=0 osc.lustre-OST0000-osc-ffff95a645874800.checksum_dump=0 osc.lustre-OST0001-osc-ffff95a645874800.checksum_dump=0 obdfilter.lustre-OST0000.checksum_dump=0 obdfilter.lustre-OST0001.checksum_dump=0 PASS 77c (48s) == sanity test 77d: checksum error on OST direct write, read ========================================================== 12:14:38 (1787760878) fail_loc=0x80000409 directio on /mnt/lustre/f77d.sanity for 8x1048576 bytes PASS fail_loc=0 fail_loc=0x80000408 directio on /mnt/lustre/f77d.sanity for 8x1048576 bytes PASS fail_loc=0 PASS 77d (12s) == sanity test 77f: repeat checksum error on write (expect error) ========================================================== 12:14:50 (1787760890) osc.lustre-OST0000-osc-ffff95a645874800.resend_count=1 osc.lustre-OST0001-osc-ffff95a645874800.resend_count=1 set checksum type to crc32, rc = 0 fail_loc=0x409 directio on /mnt/lustre/f77f.sanity.crc32 for 8x1048576 bytes Write error Input/output error (rc = -1, len = 8388608) fail_loc=0 set checksum type to adler, rc = 0 fail_loc=0x409 directio on /mnt/lustre/f77f.sanity.adler for 8x1048576 bytes Write error Input/output error (rc = -1, len = 8388608) fail_loc=0 set checksum type to crc32c, rc = 0 fail_loc=0x409 directio on /mnt/lustre/f77f.sanity.crc32c for 8x1048576 bytes Write error Input/output error (rc = -1, len = 8388608) fail_loc=0 set checksum type to t10ip512, rc = 0 fail_loc=0x409 directio on /mnt/lustre/f77f.sanity.t10ip512 for 8x1048576 bytes Write error Input/output error (rc = -1, len = 8388608) fail_loc=0 set checksum type to t10ip4K, rc = 0 fail_loc=0x409 directio on /mnt/lustre/f77f.sanity.t10ip4K for 8x1048576 bytes Write error Input/output error (rc = -1, len = 8388608) fail_loc=0 set checksum type to t10crc512, rc = 0 fail_loc=0x409 directio on /mnt/lustre/f77f.sanity.t10crc512 for 8x1048576 bytes Write error Input/output error (rc = -1, len = 8388608) fail_loc=0 set checksum type to t10crc4K, rc = 0 fail_loc=0x409 directio on /mnt/lustre/f77f.sanity.t10crc4K for 8x1048576 bytes Write error Input/output error (rc = -1, len = 8388608) fail_loc=0 set checksum type to crc32c, rc = 0 osc.lustre-OST0000-osc-ffff95a645874800.resend_count=10 osc.lustre-OST0001-osc-ffff95a645874800.resend_count=10 PASS 77f (44s) == sanity test 77g: checksum error on OST write, read ==== 12:15:35 (1787760935) fail_loc=0x8000021a 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 2.58702 s, 3.2 MB/s fail_loc=0 fail_loc=0x8000021b fail_loc=0 PASS 77g (21s) == sanity test 77k: enable/disable checksum correctly ==== 12:15:56 (1787760956) Waiting 90s for '0' Updated after 2s: want '0' got '0' remount client, checksum should be 0 Stopping client oleg316-client.virtnet /mnt/lustre (opts:) Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre remount client, checksum should be 1 Stopping client oleg316-client.virtnet /mnt/lustre (opts:) Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre remount client with option checksum, checksum should be 1 192.168.203.116@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg316-client.virtnet /mnt/lustre (opts:) Starting client: oleg316-client.virtnet: -o user_xattr,flock,checksum 192.168.203.116@tcp:/lustre /mnt/lustre remount client with option nochecksum, checksum should be 0 192.168.203.116@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg316-client.virtnet /mnt/lustre (opts:) Starting client: oleg316-client.virtnet: -o user_xattr,flock,nochecksum 192.168.203.116@tcp:/lustre /mnt/lustre Stopping client oleg316-client.virtnet /mnt/lustre (opts:) Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre Waiting 90s for '0' Updated after 2s: want '0' got '0' PASS 77k (46s) == sanity test 77l: preferred checksum type is remembered after reconnected ========================================================== 12:16:42 (1787761002) osc.lustre-OST0000-osc-ffff95a64b739000.idle_timeout=10 osc.lustre-OST0001-osc-ffff95a64b739000.idle_timeout=10 error: set_param: setting /sys/fs/lustre/osc/lustre-OST0000-osc-ffff95a64b739000/checksum_type=invalid: Invalid argument error: set_param: setting /sys/fs/lustre/osc/lustre-OST0001-osc-ffff95a64b739000/checksum_type=invalid: Invalid argument error: set_param: setting 'osc/*osc-[^mM]*/checksum_type'='invalid': Invalid argument set checksum type to invalid, rc = 22 set checksum type to crc32, rc = 0 ldlm.namespaces.*osc*.lru_size=0->400 ldlm.namespaces.*osc*.lru_max_age=600000->3900s oleg316-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff95a64b739000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff95a64b739000.ost_server_uuid in IDLE state after 0 sec oleg316-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff95a64b739000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff95a64b739000.ost_server_uuid in FULL state after 0 sec set checksum type to adler, rc = 0 ldlm.namespaces.*osc*.lru_size=0->400 ldlm.namespaces.*osc*.lru_max_age=3900000->3900s oleg316-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff95a64b739000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff95a64b739000.ost_server_uuid in IDLE state after 5 sec oleg316-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff95a64b739000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff95a64b739000.ost_server_uuid in FULL state after 0 sec set checksum type to crc32c, rc = 0 ldlm.namespaces.*osc*.lru_size=0->400 ldlm.namespaces.*osc*.lru_max_age=3900000->3900s oleg316-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff95a64b739000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff95a64b739000.ost_server_uuid in IDLE state after 6 sec oleg316-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff95a64b739000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff95a64b739000.ost_server_uuid in FULL state after 0 sec set checksum type to t10ip512, rc = 0 ldlm.namespaces.*osc*.lru_size=0->400 ldlm.namespaces.*osc*.lru_max_age=3900000->3900s oleg316-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff95a64b739000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff95a64b739000.ost_server_uuid in IDLE state after 8 sec oleg316-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff95a64b739000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff95a64b739000.ost_server_uuid in FULL state after 0 sec set checksum type to t10ip4K, rc = 0 ldlm.namespaces.*osc*.lru_size=0->400 ldlm.namespaces.*osc*.lru_max_age=3900000->3900s oleg316-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff95a64b739000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff95a64b739000.ost_server_uuid in IDLE state after 3 sec oleg316-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff95a64b739000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff95a64b739000.ost_server_uuid in FULL state after 0 sec set checksum type to t10crc512, rc = 0 ldlm.namespaces.*osc*.lru_size=0->400 ldlm.namespaces.*osc*.lru_max_age=3900000->3900s oleg316-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff95a64b739000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff95a64b739000.ost_server_uuid in IDLE state after 7 sec oleg316-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff95a64b739000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff95a64b739000.ost_server_uuid in FULL state after 0 sec set checksum type to t10crc4K, rc = 0 ldlm.namespaces.*osc*.lru_size=0->400 ldlm.namespaces.*osc*.lru_max_age=3900000->3900s oleg316-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff95a64b739000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff95a64b739000.ost_server_uuid in IDLE state after 4 sec oleg316-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff95a64b739000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff95a64b739000.ost_server_uuid in FULL state after 0 sec osc.lustre-OST0000-osc-ffff95a64b739000.idle_timeout=20 osc.lustre-OST0001-osc-ffff95a64b739000.idle_timeout=20 set checksum type to crc32c, rc = 0 PASS 77l (177s) == sanity test 77m: Verify checksum_speed is correctly read ========================================================== 12:19:39 (1787761179) checksum_speed= adler32: 428 crc32: 482 crc32c: 2751 PASS 77m (8s) == sanity test 77n: Verify read from a hole inside contiguous blocks with T10PI ========================================================== 12:19:48 (1787761188) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0105652 s, 388 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00320904 s, 1.3 MB/s SKIP: sanity test_77n f77n.sanity blocks not contiguous around hole SKIP 77n (4s) == sanity test 77o: Verify checksum_type for server (mdt and ofd(obdfilter)) ========================================================== 12:19:52 (1787761192) obdfilter.lustre-*.checksum_type: crc32 adler [crc32c] t10ip512 t10ip4K t10crc512 t10crc4K crc32 adler [crc32c] t10ip512 t10ip4K t10crc512 t10crc4K mdt.lustre-*.checksum_type: crc32 adler [crc32c] t10ip512 t10ip4K t10crc512 t10crc4K PASS 77o (13s) == sanity test 78: handle large O_DIRECT writes correctly ====================================================================== 12:20:05 (1787761205) MemFree: 2528, Max file size: 600000 MemTotal: 3693 Mem to use for directio: 1718 Smallest OST: 3759104 File size: 32 directIO rdwr round 1 of 1 directio on /mnt/lustre/f78.sanity for 32x1048576 bytes PASS PASS 78 (12s) == sanity test 79: df report consistency check =========== 12:20:18 (1787761218) sleep 5 for ZFS MDS Waiting for MDT destroys to complete UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2944 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 10240 3759104 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 10240 3759104 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 20480 7518208 1% /mnt/lustre 192.168.203.116@tcp:/lustre 7542784 20480 7518208 1% /mnt/lustre llite.lustre-ffff95a64b739000.maxbytes=18446744073709551615 lmv.lustre-clilmv-ffff95a64b739000.maxbytes=18446744073709551615 lov.lustre-clilov-ffff95a64b739000.maxbytes=18446744073709551615 mdc.lustre-MDT0000-mdc-ffff95a64b739000.maxbytes=18446744073709551615 osc.lustre-OST0000-osc-ffff95a64b739000.maxbytes=18446744073709551615 osc.lustre-OST0001-osc-ffff95a64b739000.maxbytes=18446744073709551615 llite.lustre-ffff95a64b739000.namelen_max=255 lmv.lustre-clilmv-ffff95a64b739000.namelen_max=256 lov.lustre-clilov-ffff95a64b739000.namelen_max=255 mdc.lustre-MDT0000-mdc-ffff95a64b739000.namelen_max=256 osc.lustre-OST0000-osc-ffff95a64b739000.namelen_max=255 osc.lustre-OST0001-osc-ffff95a64b739000.namelen_max=255 lod.lustre-MDT0000-mdtlov.maxbytes=18446744073709551615 lov.lustre-MDT0000-mdtlov.maxbytes=18446744073709551615 obdfilter.lustre-OST0000.maxbytes=18446744073709551615 obdfilter.lustre-OST0001.maxbytes=18446744073709551615 osc.lustre-OST0000-osc-MDT0000.maxbytes=18446744073709551615 osc.lustre-OST0001-osc-MDT0000.maxbytes=18446744073709551615 osd-zfs.lustre-MDT0000.maxbytes=18446744073709551615 osd-zfs.lustre-OST0000.maxbytes=18446744073709551615 osd-zfs.lustre-OST0001.maxbytes=18446744073709551615 osp.lustre-OST0000-osc-MDT0000.maxbytes=18446744073709551615 osp.lustre-OST0001-osc-MDT0000.maxbytes=18446744073709551615 lod.lustre-MDT0000-mdtlov.namelen_max=256 lov.lustre-MDT0000-mdtlov.namelen_max=256 obdfilter.lustre-OST0000.namelen_max=255 obdfilter.lustre-OST0001.namelen_max=255 osc.lustre-OST0000-osc-MDT0000.namelen_max=255 osc.lustre-OST0001-osc-MDT0000.namelen_max=255 osd-zfs.lustre-MDT0000.namelen_max=256 osd-zfs.lustre-OST0000.namelen_max=256 osd-zfs.lustre-OST0001.namelen_max=256 osp.lustre-OST0000-osc-MDT0000.namelen_max=255 osp.lustre-OST0001-osc-MDT0000.namelen_max=255 lod.lustre-MDT0000-mdtlov.maxbytes=18446744073709551615 lov.lustre-MDT0000-mdtlov.maxbytes=18446744073709551615 obdfilter.lustre-OST0000.maxbytes=18446744073709551615 obdfilter.lustre-OST0001.maxbytes=18446744073709551615 osc.lustre-OST0000-osc-MDT0000.maxbytes=18446744073709551615 osc.lustre-OST0001-osc-MDT0000.maxbytes=18446744073709551615 osd-zfs.lustre-MDT0000.maxbytes=18446744073709551615 osd-zfs.lustre-OST0000.maxbytes=18446744073709551615 osd-zfs.lustre-OST0001.maxbytes=18446744073709551615 osp.lustre-OST0000-osc-MDT0000.maxbytes=18446744073709551615 osp.lustre-OST0001-osc-MDT0000.maxbytes=18446744073709551615 lod.lustre-MDT0000-mdtlov.namelen_max=256 lov.lustre-MDT0000-mdtlov.namelen_max=256 obdfilter.lustre-OST0000.namelen_max=255 obdfilter.lustre-OST0001.namelen_max=255 osc.lustre-OST0000-osc-MDT0000.namelen_max=255 osc.lustre-OST0001-osc-MDT0000.namelen_max=255 osd-zfs.lustre-MDT0000.namelen_max=256 osd-zfs.lustre-OST0000.namelen_max=256 osd-zfs.lustre-OST0001.namelen_max=256 osp.lustre-OST0000-osc-MDT0000.namelen_max=255 osp.lustre-OST0001-osc-MDT0000.namelen_max=255 PASS 79 (19s) == sanity test 80: Page eviction is equally fast at high offsets too ========================================================== 12:20:37 (1787761237) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142323 s, 7.4 MB/s PASS 80 (11s) == sanity test 81a: OST should retry write when get -ENOSPC ========================================================================= 12:20:48 (1787761248) fail_loc=0x80000228