-----============= acceptance-small: sanity ============----- Sun Jun 22 07:17:23 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg402-server: /home/green/git/lustre-release/lustre/tests/except/sanity.0f.ex oleg402-server: /home/green/git/lustre-release/lustre/tests/except/sanity.ex - see client > v2_15_63-134-gdacc4b6d38 for LU-15963, go 312 - need mds1 <= 2.14.55-100-g8a84c7f9c7 for LU-14927, skip 0f - need mds1 < v2_14_55-100-g8a84c7f9c7 for LU-14927, skip 0f excepting tests: 42a 42c 42b 118c 118d 407 119i 411b skipping tests SLOW=no: 27m 60i 64b 68 71 135 136 230d 300o 842 === sanity: start setup 07:17:35 (1750591055) === oleg402-client.virtnet: executing check_config_client /mnt/lustre oleg402-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg402-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800abd69800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800abd69800.idle_timeout=debug disable quota as required oleg402-server: oleg402-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity: finish setup 07:17:53 (1750591073) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7966] preparing for tests involving mounts mke2fs 1.46.2.wc5 (26-Mar-2022) debug=all == sanity test 60a: llog_test run from kernel module and test llog_reader ========================================================== 07:17:56 (1750591076) SKIP: sanity test_60a missing subtest run-llog.sh SKIP 60a (3s) == sanity test 60b: limit repeated messages from CERROR/CWARN ========================================================== 07:17:59 (1750591079) PASS 60b (4s) == sanity test 60c: unlink file when mds full ============ 07:18:03 (1750591083) create 5000 files - open/close 879 (time 1750591095.62 total 10.01 last 87.77) - open/close 1766 (time 1750591105.63 total 20.03 last 88.57) - open/close 2528 (time 1750591115.64 total 30.04 last 76.16) - open/close 3572 (time 1750591125.64 total 40.04 last 104.35) - open/close 4590 (time 1750591135.65 total 50.05 last 101.71) total: 5000 open/close in 53.39 seconds: 93.65 ops/second fail_loc=0x80000137 - unlinked 0 (time 1750591140 ; total 0 ; last 0) total: 5000 unlinks in 37 seconds: 135.135132 unlinks/second fail_loc=0 PASS 60c (97s) == sanity test 60d: test printk console message masking == 07:19:41 (1750591181) printk=0 emerg PASS 60d (3s) == sanity test 60e: no space while new llog is being created ========================================================== 07:19:44 (1750591184) fail_loc=0x15b PASS 60e (5s) == sanity test 60f: change debug_path works ============== 07:19:49 (1750591189) debug_path=/tmp/f60f.sanity fail_loc=0x8000050e striped dir -i0 -c2 -H crush /mnt/lustre/d60f.sanity ls: cannot access /tmp/f60f.sanity*: No such file or directory 0 /tmp/f60f.sanity.1750591190.14776 debug_path=/tmp/lustre-log PASS 60f (4s) == sanity test 60g: transaction abort won't cause MDT hung ========================================================== 07:19:53 (1750591193) striped dir -i0 -c2 -H all_char /mnt/lustre/d60g.sanity /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4985: 15612 Killed ( local index=0; while true; do $LFS setdirstripe -i $(($index % $MDSCOUNT)) -c $MDSCOUNT $DIR/$tdir/subdir$index 2> /dev/null; mkdir $DIR/$tdir/subdir$index 2> /dev/null; rmdir $DIR/$tdir/subdir$index 2> /dev/null; index=$((index + 1)); done ) Started LFSCK on the device lustre-MDT0000: scrub namespace /mnt/lustre/d60g.sanity: subdir136 subdir16 subdir223 subdir230 subdir248 subdir262 subdir269 subdir286 subdir332 subdir354 subdir41 /mnt/lustre/d60g.sanity/subdir136: ls: closing directory /mnt/lustre/d60g.sanity/subdir136: No such file or directory /mnt/lustre/d60g.sanity/subdir16: /mnt/lustre/d60g.sanity/subdir223: /mnt/lustre/d60g.sanity/subdir230: /mnt/lustre/d60g.sanity/subdir248: /mnt/lustre/d60g.sanity/subdir262: /mnt/lustre/d60g.sanity/subdir269: /mnt/lustre/d60g.sanity/subdir286: /mnt/lustre/d60g.sanity/subdir332: /mnt/lustre/d60g.sanity/subdir354: /mnt/lustre/d60g.sanity/subdir41: PASS 60g (61s) == sanity test 60h: striped directory with missing stripes can be accessed ========================================================== 07:20:54 (1750591254) fail_loc=0x80000188 /home/green/git/lustre-release/lustre/tests/sanity.sh: line 10476: /mnt/lustre/d60h.sanity-0x80000188/2: No such device /home/green/git/lustre-release/lustre/tests/sanity.sh: line 10476: /mnt/lustre/d60h.sanity-0x80000188/3: No such device /home/green/git/lustre-release/lustre/tests/sanity.sh: line 10476: /mnt/lustre/d60h.sanity-0x80000188/4: No such device /home/green/git/lustre-release/lustre/tests/sanity.sh: line 10476: /mnt/lustre/d60h.sanity-0x80000188/7: No such device /home/green/git/lustre-release/lustre/tests/sanity.sh: line 10476: /mnt/lustre/d60h.sanity-0x80000188/8: No such device lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x200000400:0xc0:0x0] 0 [0:0x0:0x0] lmv_stripe_count: 3 lmv_stripe_offset: 1 lmv_hash_type: crush,migrating,fixed mdtidx FID[seq:oid:ver] 1 [0x240000400:0xbe:0x0] 0 [0x200000400:0xc0:0x0] 0 [0:0x0:0x0] /mnt/lustre/d60h.sanity-0x80000188 ~ ~ fail_loc=0x80000189 /home/green/git/lustre-release/lustre/tests/sanity.sh: line 10476: /mnt/lustre/d60h.sanity-0x80000189/2: No such device /home/green/git/lustre-release/lustre/tests/sanity.sh: line 10476: /mnt/lustre/d60h.sanity-0x80000189/3: No such device /home/green/git/lustre-release/lustre/tests/sanity.sh: line 10476: /mnt/lustre/d60h.sanity-0x80000189/4: No such device /home/green/git/lustre-release/lustre/tests/sanity.sh: line 10476: /mnt/lustre/d60h.sanity-0x80000189/7: No such device /home/green/git/lustre-release/lustre/tests/sanity.sh: line 10476: /mnt/lustre/d60h.sanity-0x80000189/8: No such device lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x200000400:0xc1:0x0] 0 [0:0x0:0x0] lmv_stripe_count: 3 lmv_stripe_offset: 1 lmv_hash_type: crush,migrating,fixed mdtidx FID[seq:oid:ver] 1 [0x240000400:0xbf:0x0] 0 [0x200000400:0xc1:0x0] 0 [0:0x0:0x0] /mnt/lustre/d60h.sanity-0x80000189 ~ ~ PASS 60h (7s) SKIP: sanity test_60i skipping SLOW test 60i == sanity test 60j: llog_reader reports corruptions ====== 07:21:02 (1750591262) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0001: clear the changelog for cl1 of all records total: 100 open/close in 1.21 seconds: 82.93 ops/second - unlinked 0 (time 1750591270 ; total 0 ; last 0) total: 100 unlinks in 0 seconds: inf unlinks/second oleg402-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg402-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps SKIP: sanity test_60j path oi.1/0x1:0xb:0x0 is not in 'O/1/d/' format lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 SKIP 60j (13s) == sanity test 61a: mmap() writes don't make sync hang ========================================================================== 07:21:15 (1750591275) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0141443 s, 290 kB/s PASS 61a (3s) == sanity test 61b: mmap() of unstriped file is successful ========================================================== 07:21:18 (1750591278) PASS 61b (3s) == sanity test 63a: Verify oig_wait interruption does not crash ================================================================= 07:21:21 (1750591281) /home/green/git/lustre-release/lustre/tests/sanity.sh: line 10610: 21143 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 10610: 21150 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 10610: 21158 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 10610: 21166 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 10610: 21174 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 10610: 21182 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 10610: 21190 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 10610: 21197 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 10610: 21205 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 10610: 21213 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k checking grant......UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3800 1283888 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2880 1284808 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1636 3514372 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3212 7119816 1% /mnt/lustre pass grant check: client:185638912 server:185638912 PASS 63a (65s) == sanity test 63b: async write errors should be returned to fsync ============================================================= 07:22:26 (1750591346) debug=-1 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0133105 s, 308 kB/s fail_loc=0x80000406 fsync: Input/output error checking grant......UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3800 1283888 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2880 1284808 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1540 3514468 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3116 7119912 1% /mnt/lustre pass grant check: client:185638912 server:185638912 PASS 63b (7s) == sanity test 64a: verify filter grant calculations (in kernel) =============================================================== 07:22:33 (1750591353) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3800 1283888 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2880 1284808 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1540 3514468 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3116 7119912 1% /mnt/lustre osc.lustre-OST0000-osc-ffff8800abd69800.cur_grant_bytes=995328 osc.lustre-OST0000-osc-ffff8800abd69800.cur_lost_grant_bytes=100655104 osc.lustre-OST0001-osc-ffff8800abd69800.cur_grant_bytes=83959808 osc.lustre-OST0001-osc-ffff8800abd69800.cur_lost_grant_bytes=28672 checking grant......UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3800 1283888 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2880 1284808 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1540 3514468 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3116 7119912 1% /mnt/lustre pass grant check: client:185638912 server:185638912 PASS 64a (5s) SKIP: sanity test_64b skipping SLOW test 64b == sanity test 64c: verify grant shrink ================== 07:22:39 (1750591359) osc.lustre-OST0000-osc-ffff8800abd69800.cur_grant_bytes=0 checking grant......UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3800 1283888 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2880 1284808 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1540 3514468 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3116 7119912 1% /mnt/lustre pass grant check: client:185638912 server:185638912 PASS 64c (4s) == sanity test 64d: check grant limit exceed ============= 07:22:43 (1750591363) 3 995328 while dd ... pid 24369 ... tgt:lustre-OST0000-osc-ffff8800abd69800 cur_grant: 995328 ?>? max_cur_granted: 498319360 tgt:lustre-OST0000-osc-ffff8800abd69800 cur_grant: 16384 ?>? max_cur_granted: 498319360 tgt:lustre-OST0000-osc-ffff8800abd69800 cur_grant: 8441856 ?>? max_cur_granted: 498319360 tgt:lustre-OST0000-osc-ffff8800abd69800 cur_grant: 8192 ?>? max_cur_granted: 498319360 tgt:lustre-OST0000-osc-ffff8800abd69800 cur_grant: 20480 ?>? max_cur_granted: 498319360 tgt:lustre-OST0000-osc-ffff8800abd69800 cur_grant: 20480 ?>? max_cur_granted: 498319360 tgt:lustre-OST0000-osc-ffff8800abd69800 cur_grant: 20480 ?>? max_cur_granted: 498319360 tgt:lustre-OST0000-osc-ffff8800abd69800 cur_grant: 20480 ?>? max_cur_granted: 498319360 tgt:lustre-OST0000-osc-ffff8800abd69800 cur_grant: 20480 ?>? max_cur_granted: 498319360 tgt:lustre-OST0000-osc-ffff8800abd69800 cur_grant: 20480 ?>? max_cur_granted: 498319360 tgt:lustre-OST0000-osc-ffff8800abd69800 cur_grant: 20480 ?>? max_cur_granted: 498319360 tgt:lustre-OST0000-osc-ffff8800abd69800 cur_grant: 20480 ?>? max_cur_granted: 498319360 tgt:lustre-OST0000-osc-ffff8800abd69800 cur_grant: 20480 ?>? max_cur_granted: 498319360 tgt:lustre-OST0000-osc-ffff8800abd69800 cur_grant: 20480 ?>? max_cur_granted: 498319360 tgt:lustre-OST0000-osc-ffff8800abd69800 cur_grant: 20480 ?>? max_cur_granted: 498319360 tgt:lustre-OST0000-osc-ffff8800abd69800 cur_grant: 20480 ?>? max_cur_granted: 498319360 tgt:lustre-OST0000-osc-ffff8800abd69800 cur_grant: 20480 ?>? max_cur_granted: 498319360 tgt:lustre-OST0000-osc-ffff8800abd69800 cur_grant: 20480 ?>? max_cur_granted: 498319360 tgt:lustre-OST0000-osc-ffff8800abd69800 cur_grant: 20480 ?>? max_cur_granted: 498319360 tgt:lustre-OST0000-osc-ffff8800abd69800 cur_grant: 20480 ?>? max_cur_granted: 498319360 tgt:lustre-OST0000-osc-ffff8800abd69800 cur_grant: 20480 ?>? max_cur_granted: 498319360 tgt:lustre-OST0000-osc-ffff8800abd69800 cur_grant: 20480 ?>? max_cur_granted: 498319360 tgt:lustre-OST0000-osc-ffff8800abd69800 cur_grant: 20480 ?>? max_cur_granted: 498319360 tgt:lustre-OST0000-osc-ffff8800abd69800 cur_grant: 20480 ?>? max_cur_granted: 498319360 tgt:lustre-OST0000-osc-ffff8800abd69800 cur_grant: 20480 ?>? max_cur_granted: 498319360 tgt:lustre-OST0000-osc-ffff8800abd69800 cur_grant: 20480 ?>? max_cur_granted: 498319360 tgt:lustre-OST0000-osc-ffff8800abd69800 cur_grant: 20480 ?>? max_cur_granted: 498319360 tgt:lustre-OST0000-osc-ffff8800abd69800 cur_grant: 20480 ?>? max_cur_granted: 498319360 tgt:lustre-OST0000-osc-ffff8800abd69800 cur_grant: 20480 ?>? max_cur_granted: 498319360 tgt:lustre-OST0000-osc-ffff8800abd69800 cur_grant: 20480 ?>? max_cur_granted: 498319360 tgt:lustre-OST0000-osc-ffff8800abd69800 cur_grant: 27394048 ?>? max_cur_granted: 498319360 tgt:lustre-OST0000-osc-ffff8800abd69800 cur_grant: 12288 ?>? max_cur_granted: 498319360 tgt:lustre-OST0000-osc-ffff8800abd69800 cur_grant: 16384 ?>? max_cur_granted: 498319360 tgt:lustre-OST0000-osc-ffff8800abd69800 cur_grant: 12288 ?>? max_cur_granted: 498319360 1000+0 records in 1000+0 records out 1048576000 bytes (1.0 GB) copied, 34.3587 s, 30.5 MB/s /home/green/git/lustre-release/lustre/tests/sanity.sh: line 10735: kill: (24369) - No such process checking grant......UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3800 1283888 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2880 1284808 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 990320 1940540 34% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 991896 5545984 16% /mnt/lustre pass grant check: client:252755968 server:252755968 Waiting for MDT destroys to complete PASS 64d (54s) == sanity test 64e: check grant consumption (no grant allocation) ========================================================== 07:23:37 (1750591417) debug=+cache Stopping client oleg402-client.virtnet /mnt/lustre (opts:) Starting client: oleg402-client.virtnet: -o user_xattr,flock 192.168.204.102@tcp:/lustre /mnt/lustre fail_loc=0x725 1+0 records in 1+0 records out 3596288 bytes (3.6 MB) copied, 0.295375 s, 12.2 MB/s fail_loc=0 Stopping client oleg402-client.virtnet /mnt/lustre (opts:) Starting client: oleg402-client.virtnet: -o user_xattr,flock 192.168.204.102@tcp:/lustre /mnt/lustre fail_loc=0x725 fail_loc=0 sanity test_64e: @@@@@@ FAIL: buf io w/o grant alloc: grants mismatch: 8192, expected 4816896 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7348:error() = /home/green/git/lustre-release/lustre/tests/sanity.sh:10756:check_grants() = /home/green/git/lustre-release/lustre/tests/sanity.sh:10819:test_64e() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7725:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7788:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7587:run_test() = /home/green/git/lustre-release/lustre/tests/sanity.sh:10821:main() Dumping lctl log to /tmp/testlogs//sanity.test_64e.*.1750591427.log rsync: chown "/tmp/testlogs/.sanity.test_64e.debug_log.oleg402-server.1750591427.log.e0ONW8" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity.test_64e.dmesg.oleg402-server.1750591427.log.QPjlAV" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 64e (14s) == sanity test 64f: check grant consumption (with grant allocation) ========================================================== 07:23:51 (1750591431) debug=+cache Stopping client oleg402-client.virtnet /mnt/lustre (opts:) Starting client: oleg402-client.virtnet: -o user_xattr,flock 192.168.204.102@tcp:/lustre /mnt/lustre 1+0 records in 1+0 records out 3825664 bytes (3.8 MB) copied, 0.132254 s, 28.9 MB/s Stopping client oleg402-client.virtnet /mnt/lustre (opts:) Starting client: oleg402-client.virtnet: -o user_xattr,flock 192.168.204.102@tcp:/lustre /mnt/lustre llite.lustre-ffff8800a74fb800.hybrid_io=0 fail_loc=0x50a fail_val=3 1+0 records in 1+0 records out 3825664 bytes (3.8 MB) copied, 0.334403 s, 11.4 MB/s sanity test_64f: @@@@@@ FAIL: buffered io, not write rpc: grants mismatch: 8445952, expected 4587520 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7348:error() = /home/green/git/lustre-release/lustre/tests/sanity.sh:10756:check_grants() = /home/green/git/lustre-release/lustre/tests/sanity.sh:10879:test_64f() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7725:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7788:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7587:run_test() = /home/green/git/lustre-release/lustre/tests/sanity.sh:10890:main() Dumping lctl log to /tmp/testlogs//sanity.test_64f.*.1750591436.log rsync: chown "/tmp/testlogs/.sanity.test_64f.debug_log.oleg402-server.1750591436.log.plYfKt" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity.test_64f.dmesg.oleg402-server.1750591436.log.16QCKB" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 64f (8s) == sanity test 64g: grant shrink on MDT ================== 07:23:59 (1750591439) 1+0 records in 1+0 records out 131072 bytes (131 kB) copied, 0.014489 s, 9.0 MB/s 1+0 records in 1+0 records out 131072 bytes (131 kB) copied, 0.0170833 s, 7.7 MB/s 1+0 records in 1+0 records out 131072 bytes (131 kB) copied, 0.0383494 s, 3.4 MB/s 1+0 records in 1+0 records out 131072 bytes (131 kB) copied, 0.0238113 s, 5.5 MB/s 1+0 records in 1+0 records out 131072 bytes (131 kB) copied, 0.0176214 s, 7.4 MB/s 1+0 records in 1+0 records out 131072 bytes (131 kB) copied, 0.0308793 s, 4.2 MB/s 1+0 records in 1+0 records out 131072 bytes (131 kB) copied, 0.0134827 s, 9.7 MB/s 1+0 records in 1+0 records out 131072 bytes (131 kB) copied, 0.0171417 s, 7.6 MB/s 1+0 records in 1+0 records out 131072 bytes (131 kB) copied, 0.0505838 s, 2.6 MB/s 1+0 records in 1+0 records out 131072 bytes (131 kB) copied, 0.0110667 s, 11.8 MB/s 0 grants, 0 pages mdc.lustre-MDT0000-mdc-ffff8800a74fb800.grant_shrink_interval=1200 mdc.lustre-MDT0001-mdc-ffff8800a74fb800.grant_shrink_interval=1200 PASS 64g (7s) == sanity test 64h: grant shrink on read ================= 07:24:07 (1750591447) osc.lustre-OST0000-osc-ffff8800a74fb800.grant_shrink=1 osc.lustre-OST0000-osc-ffff8800a74fb800.grant_shrink_interval=10 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 1.02569 s, 10.2 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.032883 s, 125 kB/s PASS 64h (14s) == sanity test 64i: shrink on reconnect ================== 07:24:20 (1750591460) 64+0 records in 64+0 records out 67108864 bytes (67 MB) copied, 4.53651 s, 14.8 MB/s fail_loc=0x80000513 fail_val=17 osc.lustre-OST0000-osc-ffff8800a74fb800.cur_grant_bytes=557056B Failing ost1 on oleg402-server Stopping /mnt/lustre-ost1 (opts:) on oleg402-server 07:24:27 (1750591467) shut down facet: ost1 facet_host: oleg402-server facet_failover_host: oleg402-server Failover ost1 to oleg402-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg402-server: oleg402-server.virtnet: executing set_default_debug all all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Started lustre-OST0000 07:24:49 (1750591489) targets are mounted 07:24:49 (1750591489) facet_failover done oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.356478 s, 23.5 MB/s PASS 64i (39s) == sanity test 64j: check grants on re-done rpc ========== 07:24:59 (1750591499) fail_loc=0x80000256 checking grant......UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3800 1283888 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2880 1284808 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1548 3605444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605444 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3124 7210888 1% /mnt/lustre pass grant check: client:76374016 server:76374016 PASS 64j (5s) == sanity test 65a: directory with no stripe info ======== 07:25:04 (1750591504) striped dir -i1 -c2 -H crush2 /mnt/lustre/d65a.sanity default stripe 1, ost count 2 PASS 65a (4s) == sanity test 65b: directory setstripe -S stripe_size*2 -i 0 -c 1 ========================================================== 07:25:08 (1750591508) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d65b.sanity dir stripe 1, default stripe 1, ost count 2 PASS 65b (4s) == sanity test 65c: directory setstripe -S stripe_size*4 -i 1 -c 1 ========================================================== 07:25:12 (1750591512) striped dir -i1 -c2 -H all_char /mnt/lustre/d65c.sanity dir stripe 1, default stripe 1, ost count 2 PASS 65c (4s) == sanity test 65d: directory setstripe -S stripe_size -c stripe_count ========================================================== 07:25:16 (1750591516) striped dir -i1 -c2 -H crush2 /mnt/lustre/d65d.sanity ll_dirstripe_verify: warning: non-sequential file1 stripe[0] 0 != file2 stripe[0] 0 dir stripe 0, default stripe 1, ost count 2 PASS 65d (3s) == sanity test 65e: directory setstripe defaults ========= 07:25:19 (1750591519) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d65e.sanity (Default) /mnt/lustre/d65e.sanity default stripe 1, ost count 2 PASS 65e (4s) == sanity test 65f: dir setstripe permission (should return error) ============================================================= 07:25:23 (1750591523) striped dir -i1 -c2 -H crush /mnt/lustre/d65f.sanityf running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [setstripe] [/mnt/lustre/d65f.sanityf] lfs setstripe: unable to open '/mnt/lustre/d65f.sanityf': Permission denied (13) PASS 65f (5s) == sanity test 65g: directory setstripe -d =============== 07:25:28 (1750591528) striped dir -i1 -c2 -H crush2 /mnt/lustre/d65g.sanity (Default) /mnt/lustre/d65g.sanity PASS 65g (4s) == sanity test 65h: directory stripe info inherit ============================================================================== 07:25:32 (1750591532) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d65h.sanity striped dir -i1 -c2 -H crush /mnt/lustre/d65h.sanity/dd1 PASS 65h (5s) == sanity test 65i: various tests to set root directory striping ========================================================== 07:25:37 (1750591537) /mnt/lustre stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/d60j.sanity stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/d65f.sanityf stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/d65g.sanity stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/f64j.sanity lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2569 0xa09 0x280000401 /mnt/lustre/f61 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2560 0xa00 0x2c0000401 /mnt/lustre/d65d.sanity stripe_count: 1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 /mnt/lustre/f63b.sanity lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2561 0xa01 0x280000401 /mnt/lustre/d65b.sanity stripe_count: 1 stripe_size: 8388608 pattern: raid0 stripe_offset: 0 /mnt/lustre/d65c.sanity stripe_count: 1 stripe_size: 16777216 pattern: raid0 stripe_offset: 1 /mnt/lustre/d65e.sanity stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/d65a.sanity stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/f61b.sanity has no stripe info /mnt/lustre/d65h.sanity stripe_count: 1 stripe_size: 8388608 pattern: raid0 stripe_offset: 0 /mnt/lustre/d60f.sanity stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/f60b.sanity lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2 0x2 0x2c0000401 /mnt/lustre lmm_fid: [0x200000007:0x1:0x0] stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/d60j.sanity lmm_fid: [0x200000402:0x1512:0x0] stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/d65f.sanityf lmm_fid: [0x240000404:0x7:0x0] stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/d65g.sanity lmm_fid: [0x240000404:0x8:0x0] stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/f64j.sanity lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000407 lmm_object_id: 0x1c lmm_fid: [0x200000407:0x1c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2569 0xa09 0x280000401 /mnt/lustre/f61 lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x1577 lmm_fid: [0x200000402:0x1577:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2560 0xa00 0x2c0000401 /mnt/lustre/d65d.sanity lmm_fid: [0x240000404:0x4:0x0] stripe_count: 1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 /mnt/lustre/f63b.sanity lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x1583 lmm_fid: [0x200000402:0x1583:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2561 0xa01 0x280000401 /mnt/lustre/d65b.sanity lmm_fid: [0x240000404:0x2:0x0] stripe_count: 1 stripe_size: 8388608 pattern: raid0 stripe_offset: 0 /mnt/lustre/d65c.sanity lmm_fid: [0x240000404:0x3:0x0] stripe_count: 1 stripe_size: 16777216 pattern: raid0 stripe_offset: 1 /mnt/lustre/d65e.sanity lmm_fid: [0x240000404:0x6:0x0] stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/d65a.sanity lmm_fid: [0x240000404:0x1:0x0] stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/f61b.sanity has no stripe info /mnt/lustre/d65h.sanity lmm_fid: [0x240000404:0x9:0x0] stripe_count: 1 stripe_size: 8388608 pattern: raid0 stripe_offset: 0 /mnt/lustre/d60f.sanity lmm_fid: [0x200000402:0x138d:0x0] stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/f60b.sanity lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x3 lmm_fid: [0x200000402:0x3:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2 0x2 0x2c0000401 PASS 65i (5s) == sanity test 65j: set default striping on root directory (bug 6367)=========================================================== 07:25:42 (1750591542) PASS 65j (5s) == sanity test 65k: validate manual striping works properly with deactivated OSCs ========================================================== 07:25:47 (1750591547) Check OST status: lustre-OST0000-osc-MDT0001 is active lustre-OST0000-osc-MDT0000 is active lustre-OST0001-osc-MDT0001 is active lustre-OST0001-osc-MDT0000 is active total: 1000 open/close in 7.79 seconds: 128.38 ops/second Deactivate: lustre-OST0000-osc-MDT0001 /home/green/git/lustre-release/lustre/utils/lfs setstripe -i 0 -c 1 /mnt/lustre/d65k.sanity/0 /home/green/git/lustre-release/lustre/utils/lfs setstripe -i 1 -c 1 /mnt/lustre/d65k.sanity/1 - unlinked 0 (time 1750591565 ; total 0 ; last 0) total: 1000 unlinks in 4 seconds: 250.000000 unlinks/second lustre-OST0000-osc-MDT0001 is Activate oleg402-server: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg402-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg402-server: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg402-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec total: 1000 open/close in 7.12 seconds: 140.35 ops/second Deactivate: lustre-OST0000-osc-MDT0000 /home/green/git/lustre-release/lustre/utils/lfs setstripe -i 0 -c 1 /mnt/lustre/d65k.sanity/0 /home/green/git/lustre-release/lustre/utils/lfs setstripe -i 1 -c 1 /mnt/lustre/d65k.sanity/1 - unlinked 0 (time 1750591588 ; total 0 ; last 0) total: 1000 unlinks in 5 seconds: 200.000000 unlinks/second lustre-OST0000-osc-MDT0000 is Activate oleg402-server: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg402-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg402-server: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg402-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec total: 1000 open/close in 7.42 seconds: 134.76 ops/second Deactivate: lustre-OST0001-osc-MDT0001 /home/green/git/lustre-release/lustre/utils/lfs setstripe -i 0 -c 1 /mnt/lustre/d65k.sanity/0 /home/green/git/lustre-release/lustre/utils/lfs setstripe -i 1 -c 1 /mnt/lustre/d65k.sanity/1 - unlinked 0 (time 1750591613 ; total 0 ; last 0) total: 1000 unlinks in 4 seconds: 250.000000 unlinks/second lustre-OST0001-osc-MDT0001 is Activate oleg402-server: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg402-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg402-server: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 oleg402-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec total: 1000 open/close in 7.28 seconds: 137.29 ops/second Deactivate: lustre-OST0001-osc-MDT0000 /home/green/git/lustre-release/lustre/utils/lfs setstripe -i 0 -c 1 /mnt/lustre/d65k.sanity/0 /home/green/git/lustre-release/lustre/utils/lfs setstripe -i 1 -c 1 /mnt/lustre/d65k.sanity/1 - unlinked 0 (time 1750591636 ; total 0 ; last 0) total: 1000 unlinks in 5 seconds: 200.000000 unlinks/second lustre-OST0001-osc-MDT0000 is Activate oleg402-server: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg402-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg402-server: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 oleg402-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec PASS 65k (102s) == sanity test 65l: lfs find on -1 stripe dir ================================================================================== 07:27:30 (1750591650) striped dir -i1 -c2 -H all_char /mnt/lustre/d65l.sanity/test_dir PASS 65l (3s) == sanity test 65m: normal user can't set filesystem default stripe ========================================================== 07:27:33 (1750591653) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [setstripe] [-c] [2] [/mnt/lustre] lfs setstripe: setstripe error for '/mnt/lustre': Operation not permitted PASS 65m (4s) == sanity test 65n: don't inherit default layout from root for new subdirectories ========================================================== 07:27:37 (1750591657) Creating new pool test_65n oleg402-server: Pool lustre.test_65n created Adding targets to pool oleg402-server: OST lustre-OST0000_UUID added to pool lustre.test_65n oleg402-server: OST lustre-OST0001_UUID added to pool lustre.test_65n /home/green/git/lustre-release/lustre/utils/lfs getstripe -d /mnt/lustre/d65n.sanity-4 stripe_count: 1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 pool: test_65n /home/green/git/lustre-release/lustre/utils/lfs getstripe -d /mnt/lustre stripe_count: 1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 pool: test_65n Destroy the created pools: test_65n lustre.test_65n oleg402-server: OST lustre-OST0000_UUID removed from pool lustre.test_65n oleg402-server: OST lustre-OST0001_UUID removed from pool lustre.test_65n oleg402-server: Pool lustre.test_65n destroyed PASS 65n (19s) == sanity test 65o: pool inheritance for mdt component === 07:27:56 (1750591676) Creating new pool test_65o oleg402-server: Pool lustre.test_65o created Adding targets to pool oleg402-server: OST lustre-OST0000_UUID added to pool lustre.test_65o oleg402-server: OST lustre-OST0001_UUID added to pool lustre.test_65o /mnt/lustre/d65o.sanity lcm_layout_gen: 0 lcm_mirror_count: 1 lcm_entry_count: 2 lcme_id: N/A lcme_mirror_id: N/A lcme_flags: 0 lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 stripe_count: 0 stripe_size: 1048576 pattern: mdt stripe_offset: -1 lcme_id: N/A lcme_mirror_id: N/A lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF stripe_count: 1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 pool: test_65o /mnt/lustre/d65o.sanity/dir2 lcm_layout_gen: 0 lcm_mirror_count: 1 lcm_entry_count: 2 lcme_id: N/A lcme_mirror_id: N/A lcme_flags: 0 lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 stripe_count: 0 stripe_size: 1048576 pattern: mdt stripe_offset: -1 lcme_id: N/A lcme_mirror_id: N/A lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF stripe_count: 1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 pool: test_65o lcm_layout_gen: 0 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: N/A lcme_mirror_id: N/A lcme_flags: 0 lcme_extent.e_start: 0 lcme_extent.e_end: EOF stripe_count: 1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 pool: test_65o Destroy the created pools: test_65o lustre.test_65o oleg402-server: OST lustre-OST0000_UUID removed from pool lustre.test_65o oleg402-server: OST lustre-OST0001_UUID removed from pool lustre.test_65o oleg402-server: Pool lustre.test_65o destroyed PASS 65o (19s) == sanity test 65p: setstripe with yaml file and huge number ========================================================== 07:28:15 (1750591695) striped dir -i1 -c2 -H crush /mnt/lustre/d65p.sanity/src_dir striped dir -i1 -c2 -H crush2 /mnt/lustre/d65p.sanity/dst_dir PASS 65p (3s) == sanity test 65q: setstripe with >=8E offset should fail ========================================================== 07:28:18 (1750591698) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d65q.sanity/src_dir lfs setstripe: lfs setstripe: cannot set default composite layout for '/mnt/lustre/d65q.sanity/src_dir': Invalid argument (22) PASS 65q (4s) == sanity test 65r: prevent all-zero offsets ============= 07:28:22 (1750591702) striped dir -i1 -c2 -H all_char /mnt/lustre/d65r.sanity PASS 65r (4s) == sanity test 66: update inode blocks count on client ========================================================================= 07:28:26 (1750591706) 8+0 records in 8+0 records out 8192 bytes (8.2 kB) copied, 0.0127061 s, 645 kB/s PASS 66 (7s) == sanity test 69: verify oa2dentry return -ENOENT doesn't LBUG ================================================================ 07:28:33 (1750591713) directio on /mnt/lustre/f69.sanity.2 for 1x4194304 bytes PASS fail_loc=0x217 directio on /mnt/lustre/f69.sanity for 2x4194304 bytes Write error No such file or directory (rc = -1, len = 8388608) fail_loc=0 directio on /mnt/lustre/f69.sanity for 2x4194304 bytes PASS directio on /mnt/lustre/f69.sanity for 1x4194304 bytes PASS fail_loc=0x217 directio on /mnt/lustre/f69.sanity for 1x4194304 bytes Read error: No such file or directory rc = -1 fail_loc=0 PASS 69 (6s) == sanity test 70a: verify health_check, health_write don't explode (on OST) ========================================================== 07:28:39 (1750591719) enable_health_write=off enable_health_write=0 enable_health_write=on enable_health_write=1 enable_health_write=0 PASS 70a (8s) SKIP: sanity test_71 skipping SLOW test 71 == sanity test 72a: Test that remove suid works properly (bug5695) ============================================================== 07:28:48 (1750591728) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7966] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/f72a.sanity] [bs=512] [count=1] 1+0 records in 1+0 records out 512 bytes (512 B) copied, 0.00671567 s, 76.2 kB/s PASS 72a (5s) == sanity test 72b: Test that we keep mode setting if without file data changed (bug 24226) ========================================================== 07:28:53 (1750591733) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7966] striped dir -i0 -c2 -H all_char /mnt/lustre/f72b.sanity-dg striped dir -i0 -c2 -H crush2 /mnt/lustre/f72b.sanity-du running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/f72b.sanity-fg] chmod: changing permissions of '/mnt/lustre/f72b.sanity-fg': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/f72b.sanity-fu] chmod: changing permissions of '/mnt/lustre/f72b.sanity-fu': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/f72b.sanity-dg] chmod: changing permissions of '/mnt/lustre/f72b.sanity-dg': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/f72b.sanity-du] chmod: changing permissions of '/mnt/lustre/f72b.sanity-du': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [2777] [/mnt/lustre/f72b.sanity-fg] chmod: changing permissions of '/mnt/lustre/f72b.sanity-fg': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [2777] [/mnt/lustre/f72b.sanity-fu] chmod: changing permissions of '/mnt/lustre/f72b.sanity-fu': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [2777] [/mnt/lustre/f72b.sanity-dg] chmod: changing permissions of '/mnt/lustre/f72b.sanity-dg': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [2777] [/mnt/lustre/f72b.sanity-du] chmod: changing permissions of '/mnt/lustre/f72b.sanity-du': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [4777] [/mnt/lustre/f72b.sanity-fg] chmod: changing permissions of '/mnt/lustre/f72b.sanity-fg': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [4777] [/mnt/lustre/f72b.sanity-fu] chmod: changing permissions of '/mnt/lustre/f72b.sanity-fu': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [4777] [/mnt/lustre/f72b.sanity-dg] chmod: changing permissions of '/mnt/lustre/f72b.sanity-dg': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [4777] [/mnt/lustre/f72b.sanity-du] chmod: changing permissions of '/mnt/lustre/f72b.sanity-du': Operation not permitted PASS 72b (5s) == sanity test 73: multiple MDC requests (should not deadlock) ========================================================== 07:28:58 (1750591738) striped dir -i1 -c2 -H crush2 /mnt/lustre/d73-1 striped dir -i1 -c2 -H crush2 /mnt/lustre/d73-2 multiop /mnt/lustre/d73-1/f73-1 vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7966 fail_loc=0x80000129 fail_loc=0 /mnt/lustre/d73-1/f73-1 has type file OK /mnt/lustre/d73-1/f73-2 has type file OK /mnt/lustre/d73-2/f73-3 has type file OK PASS 73 (30s) == sanity test 74a: ldlm_enqueue freed-export error path, ls (shouldn't LBUG) ========================================================== 07:29:28 (1750591768) fail_loc=0x8000030e /mnt/lustre/f74a fail_loc=0 PASS 74a (4s) == sanity test 74b: ldlm_enqueue freed-export error path, touch (shouldn't LBUG) ========================================================== 07:29:32 (1750591772) fail_loc=0x8000030e fail_loc=0 PASS 74b (4s) == sanity test 74c: ldlm_lock_create error path, (shouldn't LBUG) ========================================================== 07:29:36 (1750591776) fail_loc=0x319 touch: cannot touch '/mnt/lustre/f74c.sanity': No such file or directory fail_loc=0 PASS 74c (3s) == sanity test 76a: confirm clients recycle inodes properly ============================================================== 07:29:39 (1750591779) before slab objects: 89 created: 512, after slab objects: 89 PASS 76a (45s) == sanity test 76b: confirm clients recycle directory inodes properly ============================================================== 07:30:24 (1750591824) slab objects before: 89, after: 89 PASS 76b (34s) == sanity test 77a: normal checksum read/write operation ========================================================== 07:30:59 (1750591859) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.119579 s, 70.2 MB/s 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.863592 s, 9.7 MB/s PASS 77a (5s) == sanity test 77b: checksum error on client write, read ========================================================== 07:31:03 (1750591863) fail_loc=0x80000409 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 1.58401 s, 5.3 MB/s fail_loc=0 set checksum type to crc32, rc = 0 fail_loc=0x80000408 fail_loc=0 set checksum type to adler, rc = 0 fail_loc=0x80000408 fail_loc=0 set checksum type to crc32c, rc = 0 fail_loc=0x80000408 fail_loc=0 set checksum type to t10ip512, rc = 0 fail_loc=0x80000408 fail_loc=0 set checksum type to t10ip4K, rc = 0 fail_loc=0x80000408 fail_loc=0 set checksum type to t10crc512, rc = 0 fail_loc=0x80000408 fail_loc=0 set checksum type to t10crc4K, rc = 0 fail_loc=0x80000408 fail_loc=0 set checksum type to crc32c, rc = 0 PASS 77b (30s) == sanity test 77c: checksum error on client read with debug ========================================================== 07:31:33 (1750591893) debug_path=/tmp/lustre-log 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.507047 s, 16.5 MB/s osc.lustre-OST0000-osc-ffff8800a74fb800.checksum_dump=1 osc.lustre-OST0001-osc-ffff8800a74fb800.checksum_dump=1 obdfilter.lustre-OST0000.checksum_dump=1 obdfilter.lustre-OST0001.checksum_dump=1 fail_loc=0x80000408 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 2.13094 s, 3.9 MB/s fail_loc=0 osc.lustre-OST0000-osc-ffff8800a74fb800.checksum_dump=0 osc.lustre-OST0001-osc-ffff8800a74fb800.checksum_dump=0 obdfilter.lustre-OST0000.checksum_dump=0 obdfilter.lustre-OST0001.checksum_dump=0 PASS 77c (17s) == sanity test 77d: checksum error on OST direct write, read ========================================================== 07:31:50 (1750591910) fail_loc=0x80000409 directio on /mnt/lustre/f77d.sanity for 8x1048576 bytes PASS fail_loc=0 fail_loc=0x80000408 directio on /mnt/lustre/f77d.sanity for 8x1048576 bytes PASS fail_loc=0 PASS 77d (7s) == sanity test 77f: repeat checksum error on write (expect error) ========================================================== 07:31:57 (1750591917) set checksum type to crc32, rc = 0 fail_loc=0x409 directio on /mnt/lustre/f77f.sanity for 8x1048576 bytes Write error Input/output error (rc = -1, len = 8388608) fail_loc=0 set checksum type to adler, rc = 0 fail_loc=0x409 directio on /mnt/lustre/f77f.sanity for 8x1048576 bytes Write error Input/output error (rc = -1, len = 8388608) fail_loc=0 set checksum type to crc32c, rc = 0 fail_loc=0x409 directio on /mnt/lustre/f77f.sanity for 8x1048576 bytes Write error Input/output error (rc = -1, len = 8388608) fail_loc=0 set checksum type to t10ip512, rc = 0 fail_loc=0x409 directio on /mnt/lustre/f77f.sanity for 8x1048576 bytes Write error Input/output error (rc = -1, len = 8388608) fail_loc=0 set checksum type to t10ip4K, rc = 0 fail_loc=0x409 directio on /mnt/lustre/f77f.sanity for 8x1048576 bytes Write error Input/output error (rc = -1, len = 8388608) fail_loc=0 set checksum type to t10crc512, rc = 0 fail_loc=0x409 directio on /mnt/lustre/f77f.sanity for 8x1048576 bytes Write error Input/output error (rc = -1, len = 8388608) fail_loc=0 set checksum type to t10crc4K, rc = 0 fail_loc=0x409 directio on /mnt/lustre/f77f.sanity for 8x1048576 bytes Write error Input/output error (rc = -1, len = 8388608) fail_loc=0 set checksum type to crc32c, rc = 0 PASS 77f (409s) == sanity test 77g: checksum error on OST write, read ==== 07:38:46 (1750592326) fail_loc=0x8000021a 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 1.61614 s, 5.2 MB/s fail_loc=0 fail_loc=0x8000021b fail_loc=0 PASS 77g (9s) == sanity test 77k: enable/disable checksum correctly ==== 07:38:56 (1750592336) remount client, checksum should be 0 Stopping client oleg402-client.virtnet /mnt/lustre (opts:) Starting client: oleg402-client.virtnet: -o user_xattr,flock 192.168.204.102@tcp:/lustre /mnt/lustre Waiting 90s for '1' remount client, checksum should be 1 Stopping client oleg402-client.virtnet /mnt/lustre (opts:) Starting client: oleg402-client.virtnet: -o user_xattr,flock 192.168.204.102@tcp:/lustre /mnt/lustre remount client with option checksum, checksum should be 1 192.168.204.102@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg402-client.virtnet /mnt/lustre (opts:) Starting client: oleg402-client.virtnet: -o user_xattr,flock,checksum 192.168.204.102@tcp:/lustre /mnt/lustre remount client with option nochecksum, checksum should be 0 192.168.204.102@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg402-client.virtnet /mnt/lustre (opts:) Starting client: oleg402-client.virtnet: -o user_xattr,flock,nochecksum 192.168.204.102@tcp:/lustre /mnt/lustre Stopping client oleg402-client.virtnet /mnt/lustre (opts:) Starting client: oleg402-client.virtnet: -o user_xattr,flock 192.168.204.102@tcp:/lustre /mnt/lustre Waiting 90s for '0' PASS 77k (12s) == sanity test 77l: preferred checksum type is remembered after reconnected ========================================================== 07:39:09 (1750592349) osc.lustre-OST0000-osc-ffff8801309d2000.idle_timeout=10 osc.lustre-OST0001-osc-ffff8801309d2000.idle_timeout=10 error: set_param: setting /sys/fs/lustre/osc/lustre-OST0000-osc-ffff8801309d2000/checksum_type=invalid: Invalid argument error: set_param: setting /sys/fs/lustre/osc/lustre-OST0001-osc-ffff8801309d2000/checksum_type=invalid: Invalid argument error: set_param: setting 'osc/*osc-[^mM]*/checksum_type'='invalid': Invalid argument set checksum type to invalid, rc = 22 set checksum type to crc32, rc = 0 ldlm.namespaces.*osc*.lru_size=0->400 ldlm.namespaces.*osc*.lru_max_age=600000->3900s oleg402-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8801309d2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8801309d2000.ost_server_uuid in IDLE state after 0 sec oleg402-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8801309d2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8801309d2000.ost_server_uuid in FULL state after 0 sec set checksum type to adler, rc = 0 ldlm.namespaces.*osc*.lru_size=0->400 ldlm.namespaces.*osc*.lru_max_age=3900000->3900s oleg402-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8801309d2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8801309d2000.ost_server_uuid in IDLE state after 8 sec oleg402-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8801309d2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8801309d2000.ost_server_uuid in FULL state after 0 sec set checksum type to crc32c, rc = 0 ldlm.namespaces.*osc*.lru_size=0->400 ldlm.namespaces.*osc*.lru_max_age=3900000->3900s oleg402-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8801309d2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8801309d2000.ost_server_uuid in IDLE state after 12 sec oleg402-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8801309d2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8801309d2000.ost_server_uuid in FULL state after 0 sec set checksum type to t10ip512, rc = 0 ldlm.namespaces.*osc*.lru_size=0->400 ldlm.namespaces.*osc*.lru_max_age=3900000->3900s oleg402-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8801309d2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8801309d2000.ost_server_uuid in IDLE state after 13 sec oleg402-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8801309d2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8801309d2000.ost_server_uuid in FULL state after 0 sec set checksum type to t10ip4K, rc = 0 ldlm.namespaces.*osc*.lru_size=0->400 ldlm.namespaces.*osc*.lru_max_age=3900000->3900s oleg402-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8801309d2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8801309d2000.ost_server_uuid in IDLE state after 12 sec oleg402-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8801309d2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8801309d2000.ost_server_uuid in FULL state after 0 sec set checksum type to t10crc512, rc = 0 ldlm.namespaces.*osc*.lru_size=0->400 ldlm.namespaces.*osc*.lru_max_age=3900000->3900s oleg402-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8801309d2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8801309d2000.ost_server_uuid in IDLE state after 12 sec oleg402-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8801309d2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8801309d2000.ost_server_uuid in FULL state after 0 sec set checksum type to t10crc4K, rc = 0 ldlm.namespaces.*osc*.lru_size=0->400 ldlm.namespaces.*osc*.lru_max_age=3900000->3900s oleg402-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8801309d2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8801309d2000.ost_server_uuid in IDLE state after 12 sec oleg402-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8801309d2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8801309d2000.ost_server_uuid in FULL state after 0 sec osc.lustre-OST0000-osc-ffff8801309d2000.idle_timeout=20 osc.lustre-OST0001-osc-ffff8801309d2000.idle_timeout=20 set checksum type to crc32c, rc = 0 PASS 77l (129s) == sanity test 77m: Verify checksum_speed is correctly read ========================================================== 07:41:18 (1750592478) checksum_speed= adler32: 991 crc32: 1283 crc32c: 6199 PASS 77m (4s) == sanity test 77n: Verify read from a hole inside contiguous blocks with T10PI ========================================================== 07:41:22 (1750592482) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0086015 s, 476 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0015681 s, 2.6 MB/s set checksum type to t10ip512, rc = 0 1+0 records in 1+0 records out 12288 bytes (12 kB) copied, 0.0293451 s, 419 kB/s set checksum type to t10ip4K, rc = 0 1+0 records in 1+0 records out 12288 bytes (12 kB) copied, 0.0273646 s, 449 kB/s set checksum type to t10crc512, rc = 0 1+0 records in 1+0 records out 12288 bytes (12 kB) copied, 0.0140354 s, 876 kB/s set checksum type to t10crc4K, rc = 0 1+0 records in 1+0 records out 12288 bytes (12 kB) copied, 0.01539 s, 798 kB/s set checksum type to crc32c, rc = 0 PASS 77n (9s) == sanity test 77o: Verify checksum_type for server (mdt and ofd(obdfilter)) ========================================================== 07:41:31 (1750592491) obdfilter.lustre-*.checksum_type: crc32 adler crc32c t10ip512 [t10ip4K] t10crc512 t10crc4K crc32 adler crc32c t10ip512 [t10ip4K] t10crc512 t10crc4K mdt.lustre-*.checksum_type: crc32 adler crc32c t10ip512 [t10ip4K] t10crc512 t10crc4K crc32 adler crc32c t10ip512 [t10ip4K] t10crc512 t10crc4K PASS 77o (6s) == sanity test 78: handle large O_DIRECT writes correctly ====================================================================== 07:41:37 (1750592497) MemFree: 3383, Max file size: 600000 MemTotal: 3730 Mem to use for directio: 1737 Smallest OST: 3605428 File size: 32 directIO rdwr round 1 of 1 directio on /mnt/lustre/f78.sanity for 32x1048576 bytes PASS PASS 78 (5s) == sanity test 79: df report consistency check =========== 07:41:43 (1750592503) Waiting for MDT destroys to complete UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4276 1283412 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3304 1284384 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1552 3605468 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1592 3605428 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3144 7210896 1% /mnt/lustre 192.168.204.102@tcp:/lustre 7666232 3144 7210896 1% /mnt/lustre llite.lustre-ffff8801309d2000.maxbytes=17592186040320 lmv.lustre-clilmv-ffff8801309d2000.maxbytes=4402345721856 lov.lustre-clilov-ffff8801309d2000.maxbytes=17592186040320 mdc.lustre-MDT0000-mdc-ffff8801309d2000.maxbytes=4402345721856 mdc.lustre-MDT0001-mdc-ffff8801309d2000.maxbytes=4402345721856 osc.lustre-OST0000-osc-ffff8801309d2000.maxbytes=17592186040320 osc.lustre-OST0001-osc-ffff8801309d2000.maxbytes=17592186040320 llite.lustre-ffff8801309d2000.namelen_max=255 lmv.lustre-clilmv-ffff8801309d2000.namelen_max=255 lov.lustre-clilov-ffff8801309d2000.namelen_max=255 mdc.lustre-MDT0000-mdc-ffff8801309d2000.namelen_max=255 mdc.lustre-MDT0001-mdc-ffff8801309d2000.namelen_max=255 osc.lustre-OST0000-osc-ffff8801309d2000.namelen_max=255 osc.lustre-OST0001-osc-ffff8801309d2000.namelen_max=255 lod.lustre-MDT0000-mdtlov.maxbytes=4402345721856 lod.lustre-MDT0001-mdtlov.maxbytes=4402345721856 lov.lustre-MDT0000-mdtlov.maxbytes=4402345721856 lov.lustre-MDT0001-mdtlov.maxbytes=4402345721856 obdfilter.lustre-OST0000.maxbytes=17592186040320 obdfilter.lustre-OST0001.maxbytes=17592186040320 osc.lustre-OST0000-osc-MDT0000.maxbytes=17592186040320 osc.lustre-OST0000-osc-MDT0001.maxbytes=17592186040320 osc.lustre-OST0001-osc-MDT0000.maxbytes=17592186040320 osc.lustre-OST0001-osc-MDT0001.maxbytes=17592186040320 osd-ldiskfs.lustre-MDT0000.maxbytes=4402345721856 osd-ldiskfs.lustre-MDT0001.maxbytes=4402345721856 osd-ldiskfs.lustre-OST0000.maxbytes=17592186040320 osd-ldiskfs.lustre-OST0001.maxbytes=17592186040320 osp.lustre-MDT0000-osp-MDT0001.maxbytes=4402345721856 osp.lustre-MDT0001-osp-MDT0000.maxbytes=4402345721856 osp.lustre-OST0000-osc-MDT0000.maxbytes=17592186040320 osp.lustre-OST0000-osc-MDT0001.maxbytes=17592186040320 osp.lustre-OST0001-osc-MDT0000.maxbytes=17592186040320 osp.lustre-OST0001-osc-MDT0001.maxbytes=17592186040320 lod.lustre-MDT0000-mdtlov.namelen_max=255 lod.lustre-MDT0001-mdtlov.namelen_max=255 lov.lustre-MDT0000-mdtlov.namelen_max=255 lov.lustre-MDT0001-mdtlov.namelen_max=255 obdfilter.lustre-OST0000.namelen_max=255 obdfilter.lustre-OST0001.namelen_max=255 osc.lustre-OST0000-osc-MDT0000.namelen_max=255 osc.lustre-OST0000-osc-MDT0001.namelen_max=255 osc.lustre-OST0001-osc-MDT0000.namelen_max=255 osc.lustre-OST0001-osc-MDT0001.namelen_max=255 osd-ldiskfs.lustre-MDT0000.namelen_max=255 osd-ldiskfs.lustre-MDT0001.namelen_max=255 osd-ldiskfs.lustre-OST0000.namelen_max=255 osd-ldiskfs.lustre-OST0001.namelen_max=255 osp.lustre-MDT0000-osp-MDT0001.namelen_max=255 osp.lustre-MDT0001-osp-MDT0000.namelen_max=255 osp.lustre-OST0000-osc-MDT0000.namelen_max=255 osp.lustre-OST0000-osc-MDT0001.namelen_max=255 osp.lustre-OST0001-osc-MDT0000.namelen_max=255 osp.lustre-OST0001-osc-MDT0001.namelen_max=255 lod.lustre-MDT0000-mdtlov.maxbytes=4402345721856 lod.lustre-MDT0001-mdtlov.maxbytes=4402345721856 lov.lustre-MDT0000-mdtlov.maxbytes=4402345721856 lov.lustre-MDT0001-mdtlov.maxbytes=4402345721856 obdfilter.lustre-OST0000.maxbytes=17592186040320 obdfilter.lustre-OST0001.maxbytes=17592186040320 osc.lustre-OST0000-osc-MDT0000.maxbytes=17592186040320 osc.lustre-OST0000-osc-MDT0001.maxbytes=17592186040320 osc.lustre-OST0001-osc-MDT0000.maxbytes=17592186040320 osc.lustre-OST0001-osc-MDT0001.maxbytes=17592186040320 osd-ldiskfs.lustre-MDT0000.maxbytes=4402345721856 osd-ldiskfs.lustre-MDT0001.maxbytes=4402345721856 osd-ldiskfs.lustre-OST0000.maxbytes=17592186040320 osd-ldiskfs.lustre-OST0001.maxbytes=17592186040320 osp.lustre-MDT0000-osp-MDT0001.maxbytes=4402345721856 osp.lustre-MDT0001-osp-MDT0000.maxbytes=4402345721856 osp.lustre-OST0000-osc-MDT0000.maxbytes=17592186040320 osp.lustre-OST0000-osc-MDT0001.maxbytes=17592186040320 osp.lustre-OST0001-osc-MDT0000.maxbytes=17592186040320 osp.lustre-OST0001-osc-MDT0001.maxbytes=17592186040320 lod.lustre-MDT0000-mdtlov.namelen_max=255 lod.lustre-MDT0001-mdtlov.namelen_max=255 lov.lustre-MDT0000-mdtlov.namelen_max=255 lov.lustre-MDT0001-mdtlov.namelen_max=255 obdfilter.lustre-OST0000.namelen_max=255 obdfilter.lustre-OST0001.namelen_max=255 osc.lustre-OST0000-osc-MDT0000.namelen_max=255 osc.lustre-OST0000-osc-MDT0001.namelen_max=255 osc.lustre-OST0001-osc-MDT0000.namelen_max=255 osc.lustre-OST0001-osc-MDT0001.namelen_max=255 osd-ldiskfs.lustre-MDT0000.namelen_max=255 osd-ldiskfs.lustre-MDT0001.namelen_max=255 osd-ldiskfs.lustre-OST0000.namelen_max=255 osd-ldiskfs.lustre-OST0001.namelen_max=255 osp.lustre-MDT0000-osp-MDT0001.namelen_max=255 osp.lustre-MDT0001-osp-MDT0000.namelen_max=255 osp.lustre-OST0000-osc-MDT0000.namelen_max=255 osp.lustre-OST0000-osc-MDT0001.namelen_max=255 osp.lustre-OST0001-osc-MDT0000.namelen_max=255 osp.lustre-OST0001-osc-MDT0001.namelen_max=255 PASS 79 (13s) == sanity test 80: Page eviction is equally fast at high offsets too ========================================================== 07:41:55 (1750592515) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0512853 s, 20.4 MB/s PASS 80 (5s) == sanity test 81a: OST should retry write when get -ENOSPC ========================================================================= 07:42:00 (1750592520) fail_loc=0x80000228 PASS 81a (5s) == sanity test 81b: OST should return -ENOSPC when retry still fails ================================================================= 07:42:06 (1750592526) fail_loc=0x228 write: No space left on device PASS 81b (4s) == sanity test 99: cvs strange file/directory operations ========================================================== 07:42:11 (1750592531) striped dir -i1 -c2 -H crush2 /mnt/lustre/d99.sanity.cvsroot running as uid/gid/euid/egid 500/500/500/500, groups: 500 [cvs] [-d] [/mnt/lustre/d99.sanity.cvsroot] [init] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [cvs] [-d] [/mnt/lustre/d99.sanity.cvsroot] [import] [-m] [nomesg] [d99.sanity.reposname] [vtag] [rtag] cvs import: cannot open /root/.cvsignore: Permission denied N d99.sanity.reposname/README N d99.sanity.reposname/network N d99.sanity.reposname/netconsole N d99.sanity.reposname/functions No conflicts created by this import striped dir -i1 -c2 -H all_char /mnt/lustre/d99.sanity.reposname running as uid/gid/euid/egid 500/500/500/500, groups: 500 [cvs] [-d] [/mnt/lustre/d99.sanity.cvsroot] [co] [d99.sanity.reposname] cvs checkout: cannot open /root/.cvsignore: Permission denied cvs checkout: Updating d99.sanity.reposname U d99.sanity.reposname/README U d99.sanity.reposname/functions U d99.sanity.reposname/netconsole U d99.sanity.reposname/network running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [foo99] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [cvs] [add] [-m] [addmsg] [foo99] cvs add: scheduling file `foo99' for addition cvs add: use 'cvs commit' to add this file permanently running as uid/gid/euid/egid 500/500/500/500, groups: 500 [cvs] [update] cvs update: cannot open /root/.cvsignore: Permission denied cvs update: Updating . A foo99 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [cvs] [commit] [-m] [nomsg] [foo99] RCS file: /mnt/lustre/d99.sanity.cvsroot/d99.sanity.reposname/foo99,v done Checking in foo99; /mnt/lustre/d99.sanity.cvsroot/d99.sanity.reposname/foo99,v <-- foo99 initial revision: 1.1 done PASS 99 (16s) == sanity test 100: check local port using privileged port ========================================================== 07:42:27 (1750592547) PASS 100 (5s) == sanity test 101a: check read-ahead for random reads === 07:42:32 (1750592552) nreads: 10000 file size: 96MB 119.333372s, 5.49184MB/s osc.lustre-OST0000-osc-ffff8801309d2000.rpc_stats= snapshot_time: 1750592716.910515894 secs.nsecs start_time: 1750592553.415455828 secs.nsecs elapsed_time: 163.495060066 secs.nsecs read RPCs in flight: 0 write RPCs in flight: 0 DIO RPCs in flight: 0 pending write pages: 0 pending read pages: 0 read write pages per rpc rpcs % cum % | rpcs % cum % 1: 0 0 0 | 2 66 66 2: 0 0 0 | 1 33 100 read write rpcs in flight rpcs % cum % | rpcs % cum % 1: 0 0 0 | 1 33 33 2: 0 0 0 | 1 33 66 3: 0 0 0 | 1 33 100 read write offset rpcs % cum % | rpcs % cum % 0: 0 0 0 | 3 100 100 osc.lustre-OST0001-osc-ffff8801309d2000.rpc_stats= snapshot_time: 1750592716.910565078 secs.nsecs start_time: 1750592553.415475154 secs.nsecs elapsed_time: 163.495089924 secs.nsecs read RPCs in flight: 0 write RPCs in flight: 0 DIO RPCs in flight: 0 pending write pages: 0 pending read pages: 0 read write pages per rpc rpcs % cum % | rpcs % cum % 1: 0 0 0 | 2 2 2 2: 0 0 0 | 2 2 4 4: 0 0 0 | 0 0 4 8: 5 0 0 | 1 1 6 16: 6487 99 100 | 51 62 68 32: 0 0 100 | 0 0 68 64: 0 0 100 | 0 0 68 128: 0 0 100 | 0 0 68 256: 0 0 100 | 0 0 68 512: 0 0 100 | 1 1 69 1024: 0 0 100 | 25 30 100 read write rpcs in flight rpcs % cum % | rpcs % cum % 1: 6488 99 99 | 61 74 74 2: 4 0 100 | 4 4 79 3: 0 0 100 | 3 3 82 4: 0 0 100 | 10 12 95 5: 0 0 100 | 4 4 100 read write offset rpcs % cum % | rpcs % cum % 0: 5 0 0 | 5 6 6 1: 0 0 0 | 0 0 6 2: 0 0 0 | 0 0 6 4: 0 0 0 | 0 0 6 8: 0 0 0 | 0 0 6 16: 5 0 0 | 0 0 6 32: 8 0 0 | 0 0 6 64: 20 0 0 | 0 0 6 128: 29 0 1 | 0 0 6 256: 71 1 2 | 0 0 6 512: 135 2 4 | 0 0 6 1024: 268 4 8 | 3 3 9 2048: 548 8 16 | 9 10 20 4096: 1094 16 33 | 16 19 40 8192: 2157 33 66 | 34 41 81 16384: 2152 33 100 | 15 18 100 llite.lustre-ffff8801309d2000.read_ahead_stats= snapshot_time 1750592716.918570557 secs.nsecs start_time 1750592553.422822658 secs.nsecs elapsed_time 163.495747899 secs.nsecs hits 97295 samples [pages] misses 6492 samples [pages] readpage_not_consecutive 9991 samples [pages] zero_size_window 97295 samples [pages] failed_to_fast_read 6492 samples [pages] readahead_pages 6492 samples [pages] 4 15 97295 PASS 101a (168s)