-----============= acceptance-small: sanityn ============----- Mon Dec 22 02:29:17 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg148-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 02:29:46 (1766388586) === Starting client oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 Mount client oleg148-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 Started clients oleg148-client.virtnet: 192.168.201.148@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg148-client.virtnet: executing check_config_client /mnt/lustre oleg148-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg148-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9e1b82f6d000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9e1b83169800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e1b82f6d000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e1b83169800.idle_timeout=debug disable quota as required oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanityn: finish setup 02:30:28 (1766388628) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042941 s, 24.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8196] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 02:30:30 (1766388630) PASS 0 (8s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 02:30:39 (1766388639) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (7s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 02:30:45 (1766388645) -rw-r--r-- 1 root root 0 Dec 22 02:30 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (5s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 02:30:50 (1766388650) -rw-r--r-- 1 root root 0 Dec 22 02:30 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (4s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 02:30:54 (1766388654) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (5s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 02:30:59 (1766388659) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (4s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 02:31:03 (1766388663) total 0 -rwxrwxrwx 1 root root 0 Dec 22 02:30 f2a -rwxrwxrwx 1 root root 0 Dec 22 02:30 f2b total 0 -rwxrwxrwx 1 root root 0 Dec 22 02:30 f2a -rwxrwxrwx 1 root root 0 Dec 22 02:30 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.00904418 s, 56.6 kB/s PASS 2e (4s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 02:31:07 (1766388667) /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn has type file OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn has perms 0777 OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn is owned by user #500 OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn is owned by group #500 OK Can't lstat /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn: No such file or directory PASS 2f (4s) == sanityn test 2g: check blocks update on sync write ==== 02:31:11 (1766388671) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.166281 s, 12.6 MB/s /mnt/lustre/f2g.sanityn has 4096 blocks /mnt/lustre2/f2g.sanityn has 4096 blocks PASS 2g (5s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 02:31:16 (1766388676) PASS 3 (3s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 02:31:20 (1766388680) PASS 4 (6s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 02:31:25 (1766388685) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (4s) == sanityn test 6: remove of open file on other node ============================================================================ 02:31:29 (1766388689) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (5s) == sanityn test 7: remove of open directory on other node ======================================================================= 02:31:34 (1766388694) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (4s) == sanityn test 8: remove of open special file on other node ==================================================================== 02:31:38 (1766388698) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (4s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 02:31:42 (1766388702) PASS 9a (5s) == sanityn test 9b: append to striped sparse file ======== 02:31:47 (1766388707) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358595 s, 29.2 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0536304 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (4s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 02:31:51 (1766388711) 1+0 records in 1+0 records out 1 byte copied, 0.00696465 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0097204 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00694849 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0116799 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0098084 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0103044 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00655474 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0108318 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00836799 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0171359 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0144131 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0117828 s, 0.1 kB/s PASS 10a (4s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 02:31:55 (1766388715) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00703599 s, 437 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0360264 s, 114 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000410074 s, 7.5 MB/s PASS 10b (3s) == sanityn test 11: execution of file opened for write should return error ============================================================== 02:31:58 (1766388718) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d11 multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8196 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 300: /mnt/lustre2/d11/f: Text file busy PASS 11 (4s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 02:32:02 (1766388722) warning: '-runtime' deprecated, use '-t runtime' instead - link 818 (time 1766388733.38 total 10.01 last 81.70) - link 1653 (time 1766388743.39 total 20.02 last 83.45) - link 2211 (time 1766388753.40 total 30.03 last 55.74) - link 2873 (time 1766388763.41 total 40.04 last 66.10) - link 3369 (time 1766388773.42 total 50.05 last 49.57) - link 3918 (time 1766388783.43 total 60.06 last 54.83) - link 4571 (time 1766388793.44 total 70.07 last 65.24) - link 5184 (time 1766388803.45 total 80.08 last 61.27) - link 5754 (time 1766388813.47 total 90.10 last 56.88) total: 6372 link in 100.00 seconds: 63.72 ops/second using seed 2460287429 running for 100 seconds - stat 10000 (time 1766388730 ; total 6 ; last 6) - stat 20000 (time 1766388736 ; total 12 ; last 6) - stat 30000 (time 1766388741 ; total 17 ; last 5) - stat 40000 (time 1766388747 ; total 23 ; last 6) - stat 50000 (time 1766388753 ; total 29 ; last 6) - stat 60000 (time 1766388759 ; total 35 ; last 6) - stat 70000 (time 1766388766 ; total 42 ; last 7) - stat 80000 (time 1766388772 ; total 48 ; last 6) - stat 88133 (time 1766388783 ; total 59 ; last 11) - stat 90000 (time 1766388785 ; total 61 ; last 2) - stat 96368 (time 1766388796 ; total 72 ; last 11) - stat 100000 (time 1766388801 ; total 77 ; last 5) - stat 105996 (time 1766388812 ; total 88 ; last 11) - stat 110000 (time 1766388818 ; total 94 ; last 6) total: 113344 stats in 100 seconds: 1133.439941 stats/second - unlinked 0 (time 1766388774 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile6372) error: No such file or directory total: 6371 unlinks in 98 seconds: 65.010201 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (22663) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (22665) - No such process PASS 12 (153s) == sanityn test 13: test directory page revocation ======= 02:34:35 (1766388875) striped dir -i1 -c2 -H crush2 /mnt/lustre/d13 f13 f13 PASS 13 (6s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 02:34:41 (1766388881) striped dir -i0 -c2 -H crush /mnt/lustre/d14aa.sanityn multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.8196 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 335: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 23874 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (4s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 02:34:45 (1766388885) striped dir -i0 -c2 -H all_char /mnt/lustre/d14ab.sanityn open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 24464 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (4s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 02:34:49 (1766388889) striped dir -i0 -c2 -H all_char /mnt/lustre/d14b.sanityn truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 350: 25054 Terminated $DIR1/$tdir/sleep 60 PASS 14b (5s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 02:34:54 (1766388894) striped dir -i0 -c2 -H crush2 /mnt/lustre/d14c.sanityn cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 362: 25648 Terminated $DIR1/$tdir/sleep 60 PASS 14c (4s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 02:34:58 (1766388898) striped dir -i0 -c2 -H crush /mnt/lustre/d14d.sanityn chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 373: 26241 Terminated $DIR1/$tdir/sleep 60 PASS 14d (5s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 02:35:04 (1766388904) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky Waiting for MDT destroys to complete 0 STRIPECOUNT=2 ORIGFREE=7207404 MAXFREE=2097152000 0 lustre-OST0000 avl=440 grnt=495 diff=-55 limit=46331 FULL lustre-OST0001 avl=1460 grnt=1639 diff=-179 limit=46331 FULL ERROR: dd blocks 7198202 != file size 7198224 LOG LOG2 file /tmp/oos2.log:dd: error writing '/mnt/lustre/oosfile': No space left on device /tmp/oos2.log:3606650+0 records in /tmp/oos2.log:3606649+0 records out /tmp/oos2.log:3693208576 bytes (3.7 GB, 3.4 GiB) copied, 501.216 s, 7.4 MB/s /tmp/oos2.log2:dd: error writing '/mnt/lustre2/oosfile2': No space left on device /tmp/oos2.log2:3591554+0 records in /tmp/oos2.log2:3591553+0 records out /tmp/oos2.log2:3677750272 bytes (3.7 GB, 3.4 GiB) copied, 500.789 s, 7.3 MB/s Waiting for MDT destroys to complete Waiting for MDT destroys to complete PASS 15 (545s) == sanityn test 16a: 2500 iterations of dual-mount fsx === 02:44:09 (1766389449) keep default fallocate mode: 0 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.118092 s, 71.0 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.126737 s, 66.2 MB/s Chance of close/open is 1 in 50 Seed set to 9451 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn skipping zero length zero range skipping zero size read fallocating to largest ever: 0x5dfdd5 fallocating to largest ever: 0x85a29b truncating to largest ever: 0x866a65 000100[0] 1766389453.777643 write 0x7ee6d9 thru 0x7f3c3e (0x5566 bytes) truncating to largest ever: 0x9f7658 000200[0] 1766389456.466787 trunc from 0x6e9bd4 to 0x53e126 (0x1abaad bytes) 000300[0] 1766389458.759367 punch from 0x5628eb to 0x56a082 (0x7798 bytes) 000400[1] 1766389460.681357 trunc from 0x867fe0 to 0x876d6c (0xed8d bytes) fallocating to largest ever: 0xa00000 000500[1] 1766389462.489481 mapread 0x185ea1 thru 0x18d887 (0x79e7 bytes) 000600[1] 1766389464.321944 read 0x04578f thru 0x04ac0a (0x547c bytes) 000700[1] 1766389466.680342 fallocate 0x4677e9 thru 0x47191a (0xa132 bytes) 000800[1] 1766389468.965939 punch from 0x3ae1d5 to 0x3be0b1 (0xfedd bytes) 001000[0] 1766389473.248829 mapwrite 0x098587 thru 0x09cb76 (0x45f0 bytes) 001300[1] 1766389479.532737 trunc from 0x1038c6 to 0x10c6c2 (0x8dfd bytes) 001400[1] 1766389481.893687 mapread 0x0856bd thru 0x08d42e (0x7d72 bytes) 001700[0] 1766389488.683256 punch from 0x1a82a9 to 0x1b2d93 (0xaaeb bytes) 001800[0] 1766389491.051407 mapwrite 0x6ae864 thru 0x6b5d5c (0x74f9 bytes) 001900[0] 1766389492.943494 read 0x154214 thru 0x15e92d (0xa71a bytes) 002000[1] 1766389495.167947 read 0x0a9132 thru 0x0b01f4 (0x70c3 bytes) 002100[0] 1766389497.206969 mapread 0x3df3d8 thru 0x3e0474 (0x109d bytes) 002200[1] 1766389499.385754 punch from 0x18c43a to 0x192b07 (0x66ce bytes) 002300[1] 1766389501.593491 write 0x23912e thru 0x2431f5 (0xa0c8 bytes) 002400[1] 1766389503.536593 fallocate 0x5c0bb1 thru 0x5c1b73 (0xfc3 bytes) 002500[0] 1766389505.601609 fallocate 0x3490a9 thru 0x35451e (0xb476 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 9505 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn truncating to largest ever: 0x64f345 truncating to largest ever: 0x98023c fallocating to largest ever: 0x9caaba 000100[1] 1766389507.746459 read 0x5d6000 thru 0x5e55bb (0xf5bc bytes) 000200[0] 1766389509.733773 read 0x4c1000 thru 0x4c2579 (0x157a bytes) 000300[1] 1766389511.831613 trunc from 0x531eea to 0x53c689 (0xa7a0 bytes) 000400[0] 1766389513.966370 trunc from 0x3fac1a to 0x3fe78d (0x3b74 bytes) fallocating to largest ever: 0x9cda4f 000600[0] 1766389518.325336 write_OD 0x84b000 thru 0x850fff (0x6000 bytes) 000800[0] 1766389522.124171 mapread 0x068000 thru 0x06afbb (0x2fbc bytes) 000900[1] 1766389524.167442 trunc from 0x5fe7b2 to 0x9b07b2 (0x3b2001 bytes) 001000[0] 1766389526.359702 trunc from 0x550efe to 0x55ee6f (0xdf72 bytes) 001100[1] 1766389528.091437 trunc from 0x883bde to 0x99f299 (0x11b6bc bytes) 001300[0] 1766389531.166145 mapwrite 0x39f000 thru 0x3ac4d8 (0xd4d9 bytes) fallocating to largest ever: 0x9dc677 001400[1] 1766389532.670644 mapread 0x065000 thru 0x071084 (0xc085 bytes) 001500[1] 1766389534.113633 fallocate 0x112ac8 thru 0x11e23b (0xb774 bytes) truncating to largest ever: 0x9e21eb 001600[1] 1766389535.705669 mapwrite 0x9fc000 thru 0x9ffba8 (0x3ba9 bytes) fallocating to largest ever: 0x9fce31 001700[1] 1766389537.337065 mapread 0x5ae000 thru 0x5bc974 (0xe975 bytes) 001800[1] 1766389539.012258 mapread 0x147000 thru 0x155910 (0xe911 bytes) 001900[1] 1766389541.042006 trunc from 0x5e25e0 to 0x5e96a5 (0x70c6 bytes) 002100[0] 1766389544.257738 fallocate 0x35df91 thru 0x36dbd7 (0xfc47 bytes) 002200[1] 1766389545.860674 trunc from 0x01fba3 to 0x028db1 (0x920f bytes) truncating to largest ever: 0x9fdc76 All operations completed A-OK! PASS 16a (105s) == sanityn test 16b: 2500 iterations of dual-mount fsx at small size ========================================================== 02:45:53 (1766389553) keep default fallocate mode: 0 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.100669 s, 83.3 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.097352 s, 86.2 MB/s Chance of close/open is 1 in 50 Seed set to 9554 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn truncating to largest ever: 0x8e7107 skipping zero size read fallocating to largest ever: 0x9285ba 000100[0] 1766389556.578952 trunc from 0x2e0811 to 0x48539a (0x1a4b8a bytes) 000200[1] 1766389558.053443 trunc from 0x8c7c75 to 0x288d15 (0x63ef5f bytes) truncating to largest ever: 0x92b785 000300[0] 1766389559.575427 mapread 0x8602e1 thru 0x860669 (0x389 bytes) truncating to largest ever: 0x9f31f1 000400[0] 1766389561.282333 mapread 0x5825c0 thru 0x584347 (0x1d88 bytes) 000500[1] 1766389562.718499 punch from 0x21224b to 0x212cde (0xa94 bytes) 000600[0] 1766389564.263573 write 0x46c48d thru 0x46c895 (0x409 bytes) 000700[1] 1766389565.937156 mapwrite 0x49a054 thru 0x49a929 (0x8d6 bytes) truncating to largest ever: 0x9fa3cb 000900[0] 1766389569.724588 punch from 0x4cddbb to 0x4ce791 (0x9d7 bytes) 001000[1] 1766389571.386707 mapwrite 0x88c49e thru 0x88d106 (0xc69 bytes) 001100[0] 1766389573.043213 mapwrite 0x045025 thru 0x046a85 (0x1a61 bytes) 001200[1] 1766389574.881004 trunc from 0x2bc730 to 0x393a42 (0xd7313 bytes) 001300[1] 1766389576.571879 mapwrite 0x6a1d16 thru 0x6a3811 (0x1afc bytes) 001500[1] 1766389579.857350 mapread 0x4e4d6b thru 0x4e6539 (0x17cf bytes) 001600[0] 1766389581.391052 write 0x24fe57 thru 0x2507dc (0x986 bytes) 001700[0] 1766389582.969992 fallocate 0x3cbbee thru 0x3cc8af (0xcc2 bytes) 001800[0] 1766389584.583276 trunc from 0x9c70d5 to 0x9c776a (0x696 bytes) 001900[0] 1766389586.075308 trunc from 0x644f4e to 0x918098 (0x2d314b bytes) 002000[1] 1766389587.692352 write 0x5350f3 thru 0x5367c7 (0x16d5 bytes) 002100[0] 1766389589.294753 read 0x0e9ba7 thru 0x0eb4cd (0x1927 bytes) 002200[1] 1766389590.801812 fallocate 0x7a2d93 thru 0x7a33d8 (0x646 bytes) 002400[1] 1766389593.735810 fallocate 0x9a64e9 thru 0x9a8188 (0x1ca0 bytes) All operations completed A-OK! PASS 16b (44s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 02:46:37 (1766389597) keep default fallocate mode: 0 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0919605 s, 91.2 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.095255 s, 88.1 MB/s Waiting for MDT destroys to complete Chance of close/open is 1 in 50 Seed set to 9610 fd 0: /mnt/lustre/f16c.sanityn fd 1: /mnt/lustre2/f16c.sanityn fallocating to largest ever: 0x44ff85 skipping zero length zero range fallocating to largest ever: 0x8b9416 fallocating to largest ever: 0x8bad1e fallocating to largest ever: 0x955ed6 000100[1] 1766389612.799099 write 0x2c9d54 thru 0x2d7eb0 (0xe15d bytes) truncating to largest ever: 0x9d5b87 000200[1] 1766389614.916059 mapwrite 0x72469d thru 0x72e5a0 (0x9f04 bytes) truncating to largest ever: 0x9d8751 000300[0] 1766389617.128432 read 0x542cf0 thru 0x547604 (0x4915 bytes) 000400[1] 1766389619.252277 write 0x2f11d6 thru 0x2f4aac (0x38d7 bytes) 000500[0] 1766389621.552999 read 0x77f4c7 thru 0x787a39 (0x8573 bytes) 000600[0] 1766389623.524741 trunc from 0x88db33 to 0x0e146e (0x7ac6c4 bytes) 000700[1] 1766389625.756680 mapwrite 0x320068 thru 0x325e0e (0x5da7 bytes) 000800[0] 1766389627.839996 read 0x2ef446 thru 0x2f1a17 (0x25d2 bytes) 000900[0] 1766389630.014506 fallocate 0x0adf84 thru 0x0b9a6a (0xbae7 bytes) 001000[1] 1766389632.196489 write 0x12d9a4 thru 0x134bf3 (0x7250 bytes) 001200[1] 1766389635.952173 punch from 0x38a416 to 0x38c055 (0x1c40 bytes) fallocating to largest ever: 0xa00000 001300[0] 1766389637.606004 mapread 0x19dc81 thru 0x1a7761 (0x9ae1 bytes) 001400[1] 1766389639.382000 mapread 0x23c4c9 thru 0x23f170 (0x2ca8 bytes) 001500[0] 1766389641.271438 trunc from 0x848662 to 0x568c47 (0x2dfa1a bytes) 001600[1] 1766389643.048131 trunc from 0x801f5e to 0x80e0b2 (0xc155 bytes) 001700[0] 1766389644.937204 mapwrite 0x5f5d99 thru 0x5ff10f (0x9377 bytes) 001800[0] 1766389646.816995 trunc from 0x5dda52 to 0x1c9a17 (0x41403a bytes) 001900[1] 1766389648.642491 trunc from 0x6d4317 to 0x6de2be (0x9fa8 bytes) 002000[1] 1766389650.397829 mapread 0x3a8ad2 thru 0x3aa3d2 (0x1901 bytes) 002100[1] 1766389652.539274 fallocate 0x3cf091 thru 0x3d6351 (0x72c1 bytes) 002200[0] 1766389654.434697 punch from 0x034fe0 to 0x03b255 (0x6276 bytes) 002300[1] 1766389656.404243 mapwrite 0x155c49 thru 0x15c1d0 (0x6588 bytes) 002400[1] 1766389658.621551 mapread 0x356d79 thru 0x35b3e6 (0x466e bytes) 002500[1] 1766389660.282601 punch from 0x85f9d1 to 0x8697b9 (0x9de9 bytes) All operations completed A-OK! PASS 16c (66s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 02:47:43 (1766389663) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0926251 s, 90.6 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0818995 s, 102 MB/s ldlm.namespaces.MGC192.168.201.148@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9e1b82f6d000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9e1b83169800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9e1b82f6d000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9e1b83169800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9e1b82f6d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9e1b83169800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9e1b82f6d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9e1b83169800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.40169 s, 43.7 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.15821 s, 90.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.17744 s, 89.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.19256 s, 47.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.41044 s, 74.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.53147 s, 68.5 MB/s PASS 16d (15s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 02:47:58 (1766389678) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.131451 s, 79.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186199 s, 56.3 MB/s PASS 16e (3s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 02:48:01 (1766389681) TIMEOUT PASS 16f (28s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 02:48:29 (1766389709) TIMEOUT PASS 16g (32s) == sanityn test 16h: mmap read after truncate file ======= 02:49:01 (1766389741) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d16h.sanityn ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (11s) == sanityn test 16i: read after truncate file ============ 02:49:12 (1766389752) striped dir -i0 -c2 -H all_char /mnt/lustre/d16i.sanityn ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (11s) == sanityn test 16j: race dio with buffered i/o ========== 02:49:24 (1766389764) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.77604 s, 14.8 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.57893 s, 10.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.69137 s, 9.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.71478 s, 15.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.36008 s, 11.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.93107 s, 13.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.88296 s, 9.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.77242 s, 14.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.45181 s, 10.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.59225 s, 10.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.84147 s, 9.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.11564 s, 23.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.31276 s, 20.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.78304 s, 14.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.93522 s, 13.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.29847 s, 20.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.39196 s, 18.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.98391 s, 13.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.96556 s, 13.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (38s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 02:50:02 (1766389802) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fallocating to largest ever: 0x3428364 skipping zero size read fallocating to largest ever: 0x3428364 skipping zero size read fallocating to largest ever: 0x3428364 skipping zero size read fallocating to largest ever: 0x3428364 skipping zero size read fallocating to largest ever: 0x3428364 skipping zero size read truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db fallocating to largest ever: 0x3ddbbf6 fallocating to largest ever: 0x3ddbbf6 fallocating to largest ever: 0x3ddbbf6 fallocating to largest ever: 0x3ddbbf6 fallocating to largest ever: 0x3ddbbf6 fallocating to largest ever: 0x3fa9cee fallocating to largest ever: 0x3fa9cee fallocating to largest ever: 0x3fa9cee fallocating to largest ever: 0x3fa9cee fallocating to largest ever: 0x3fa9cee truncating to largest ever: 0x3fb80f5 truncating to largest ever: 0x3fb80f5 truncating to largest ever: 0x3fb80f5 truncating to largest ever: 0x3fb80f5 truncating to largest ever: 0x3fb80f5 All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Mon Dec 22 02:50:31 EST 2025 + fsx 2 done: rc=0 Mon Dec 22 02:50:31 EST 2025 + fsx 3 done: rc=0 Mon Dec 22 02:50:31 EST 2025 + fsx 4 done: rc=0 Mon Dec 22 02:50:31 EST 2025 + fsx 5 done: rc=0 Mon Dec 22 02:50:31 EST 2025 PASS 16k (37s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 02:50:39 (1766389839) fail_loc=0x8000030a PASS 17 (8s) == sanityn test 18: mmap sanity check =========================================================================================== 02:50:47 (1766389847) mmap test1: basic mmap operation (PASS, 0.13872s) mmap test2: MAP_PRIVATE not write back (PASS, 0.11716s) mmap test3: concurrent mmap ops on two nodes (PASS, 26.638s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.4337s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.17056s) mmap test6: check mmap write/read content on two nodes (PASS, 0.20212s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 0.88017s) mmap test11: multi-thread shared mmap (PASS, 1.1497s) mmap test10: mtime not change for readonly mmap access (PASS, 1.0618s) PASS 18 (42s) == sanityn test 19: test concurrent uncached read races ========================================================================= 02:51:29 (1766389889) 32+0 records in 32+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.3046 s, 55.1 MB/s loop 5 loop 10 loop 15 loop 20 PASS 19 (20s) == sanityn test 20: test extra readahead page left in cache ============================================================== 02:51:49 (1766389909) striped dir -i0 -c2 -H crush2 /mnt/lustre/d20.sanityn PASS 20 (5s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 02:51:54 (1766389914) striped dir -i1 -c2 -H crush2 /mnt/lustre/d21.sanityn rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (4s) == sanityn test 23: others should see updated atime while another read============================================================== 02:51:58 (1766389918) now is 1766389920 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.8196 new atime is 1766389981 PASS 23 (67s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 02:53:05 (1766389985) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2036 1269268 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 8508 3598512 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 30308 3576712 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 38816 7175224 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2036 1269268 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1788 1269516 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 8508 3598512 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 30308 3576712 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 38816 7175224 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1000.0K 354 999.7K 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1000.0K 279 999.7K 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 256.0K 402 255.6K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 256.0K 403 255.6K 1% /mnt/lustre[OST:1] filesystem_summary: 511.8K 633 511.2K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1000.0K 354 999.7K 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1000.0K 279 999.7K 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 256.0K 402 255.6K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 256.0K 403 255.6K 1% /mnt/lustre2[OST:1] filesystem_summary: 511.8K 633 511.2K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 1.3G 2.0M 1.2G 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1.3G 1.7M 1.2G 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3.7G 8.3M 3.4G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.7G 29.6M 3.4G 1% /mnt/lustre[OST:1] filesystem_summary: 7.3G 37.9M 6.8G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1024000 354 1023646 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1024000 279 1023721 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 262144 402 261742 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 262144 403 261741 1% /mnt/lustre2[OST:1] filesystem_summary: 524116 633 523483 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2036 1269268 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 8508 3598512 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 30308 3576712 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 38816 7175224 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1000.0K 354 999.7K 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1000.0K 279 999.7K 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 256.0K 402 255.6K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 256.0K 403 255.6K 1% /mnt/lustre2[OST:1] filesystem_summary: 511.8K 633 511.2K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1024000 354 1023646 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1024000 279 1023721 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 262144 402 261742 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 262144 403 261741 1% /mnt/lustre[OST:1] filesystem_summary: 524116 633 523483 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1024000 354 1023646 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1024000 279 1023721 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 262144 402 261742 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 262144 403 261741 1% /mnt/lustre2[OST:1] filesystem_summary: 524116 633 523483 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2036 1269268 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 8508 3598512 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 30308 3576712 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 38816 7175224 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2036 1269268 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1788 1269516 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 8508 3598512 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 30308 3576712 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 38816 7175224 1% /mnt/lustre2 PASS 24a (6s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 02:53:11 (1766389991) PASS 24b (6s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 02:53:17 (1766389997) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (5s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 02:53:22 (1766390002) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] Can't lstat /mnt/lustre2/d25b.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] Can't lstat /mnt/lustre2/d25b.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] PASS 25b (6s) == sanityn test 26a: allow mtime to get older ============ 02:53:29 (1766390009) utime: good mknod times 1766390009 <= 1766390010 <= 1766390010 for /mnt/lustre/f26a utime: good mknod times 1766390009 <= 1766390010 <= 1766390010 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (6s) == sanityn test 26b: sync mtime between ost and mds ====== 02:53:35 (1766390015) PASS 26b (6s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 02:53:41 (1766390021) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8196 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205289279615 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2025-12-22 02:53:42.000000000 -0500 Birth: 2025-12-22 02:53:43.000000000 -0500 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205289279615 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2025-12-22 02:53:42.000000000 -0500 Birth: 2025-12-22 02:53:43.000000000 -0500 PASS 26c (7s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 02:53:49 (1766390029) striped dir -i0 -c2 -H crush2 /mnt/lustre/d30.sanityn 0 PASS 30 (8s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 02:53:57 (1766390037) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d31a.sanityn fail_loc=0x314 PASS 31a (7s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 02:54:04 (1766390044) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg148-server mds-ost sync done. Waiting for MDT destroys to complete striped dir -i1 -c2 -H crush2 /mnt/lustre/d31b.sanityn fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.148@tcp:/lustre 7666232 39944 7174096 1% /mnt/lustre2 PASS 31b (13s) == sanityn test 31r: open-rename(replace) race =========== 02:54:17 (1766390057) -rw-r--r-- 1 root root 0 Dec 22 02:54 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (8s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 02:54:25 (1766390065) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1207: 47630 Terminated tail -f $tfile PASS 31s (6s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 02:54:31 (1766390071) PASS 31t (6s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 02:54:38 (1766390078) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (2s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 02:54:40 (1766390080) Stopping /mnt/lustre-mds1 (opts:) on oleg148-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 create file on 2nd stripe under striped directory /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 2 /mnt/lustre/d33c.sanityn/striped touch /mnt/lustre2/d33c.sanityn/striped/subfile SoLC count 2 Below operations shouldn't trigger Solc: create file on 2nd stripe after setattr chmod 777 /mnt/lustre/d33c.sanityn/striped; touch /mnt/lustre2/d33c.sanityn/striped/subfile create remote dir and local dir /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33c.sanityn/remote create local dir after remote dir creation transaction commit /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33c.sanityn/remote.3; do_facet mds2 /home/green/git/lustre-release/lustre/utils/lctl set_param -n osd*.*MDT0001.force_sync 1; mkdir /mnt/lustre/d33c.sanityn/local.3 Solc count 0 PASS 33c (33s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 02:55:14 (1766390114) remote directory create /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre2/d33d.sanityn/subdir CoS count 2 cross-MDT rename mkdir /mnt/lustre/d33d.sanityn; /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn/d1; /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33d.sanityn/d2; touch /mnt/lustre/d33d.sanityn/d1/src mv /mnt/lustre2/d33d.sanityn/d1/src /mnt/lustre2/d33d.sanityn/d2/tgt CoS count 1 migrate /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn /home/green/git/lustre-release/lustre/utils/lfs migrate -m 1 /mnt/lustre2/d33d.sanityn CoS count 2 PASS 33d (30s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 02:55:44 (1766390144) plain dir creation mkdir /mnt/lustre2/d33e.sanityn/plain open file and write echo abc > /mnt/lustre2/d33e.sanityn/f33e.sanityn append write echo dfg >> /mnt/lustre2/d33e.sanityn/f33e.sanityn setattr touch /mnt/lustre2/d33e.sanityn/f33e.sanityn file unlink rm /mnt/lustre2/d33e.sanityn/f33e.sanityn plain dir unlink rmdir /mnt/lustre2/d33e.sanityn/plain striped directory creation /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 2 /mnt/lustre2/d33e.sanityn/striped set default LMV to create striped subdir /home/green/git/lustre-release/lustre/utils/lfs setdirstripe -D -c 2 /mnt/lustre/d33e.sanityn striped subdir creation createmany -d /mnt/lustre/d33e.sanityn/subdir 100 total: 100 mkdir in 1.08 seconds: 92.72 ops/second sub file creation and write createmany -o /mnt/lustre/d33e.sanityn/subfile 100; echo abc > /mnt/lustre/d33e.sanityn/subfile1 total: 100 open/close in 1.17 seconds: 85.25 ops/second sub file append write echo dfg >> /mnt/lustre2/d33e.sanityn/subfile2 subdir setatttr touch /mnt/lustre2/d33e.sanityn/subdir1 subdir unlink unlinkmany -d /mnt/lustre/d33e.sanityn/subdir 100 - unlinked 0 (time 1766390148 ; total 0 ; last 0) total: 100 unlinks in 4 seconds: 25.000000 unlinks/second sub file unlink unlinkmany /mnt/lustre2/d33e.sanityn/subfile 100 - unlinked 0 (time 1766390153 ; total 1 ; last 1) total: 100 unlinks in 2 seconds: 50.000000 unlinks/second sub file creation follows striped dir chmod chmod 777 /mnt/lustre/d33e.sanityn/striped; touch /mnt/lustre/d33e.sanityn/striped/subfile striped directory unlink rm -rf /mnt/lustre2/d33e.sanityn/striped directory unlink rm -rf /mnt/lustre2/d33e.sanityn CoS count 0 PASS 33e (15s) == sanityn test 34: no lock timeout under IO ============= 02:55:58 (1766390158) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg148-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9e1b82f6d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e1b82f6d000.ost_server_uuid in FULL state after 0 sec oleg148-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e1b82f6d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e1b82f6d000.ost_server_uuid in IDLE state after 0 sec oleg148-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9e1b82f6d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e1b82f6d000.ost_server_uuid in FULL state after 0 sec oleg148-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e1b82f6d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e1b82f6d000.ost_server_uuid in IDLE state after 0 sec oleg148-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9e1b82f6d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e1b82f6d000.ost_server_uuid in FULL state after 0 sec oleg148-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e1b82f6d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e1b82f6d000.ost_server_uuid in IDLE state after 0 sec PASS 34 (101s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 02:57:39 (1766390259) striped dir -i1 -c2 -H all_char /mnt/lustre/d35.sanityn fail_loc=0x80000317 Race attempt 0 Wait for 57620 57706 for 60 sec... - open/close 924 (time 1766390273.82 total 10.00 last 92.37) - open/close 1827 (time 1766390283.82 total 20.00 last 90.29) - open/close 2903 (time 1766390293.84 total 30.02 last 107.44) - open/close 3882 (time 1766390303.85 total 40.03 last 97.78) total: 4000 open/close in 41.04 seconds: 97.47 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.201.148@tcp:/lustre 7.4G 40M 6.9G 1% /mnt/lustre 192.168.201.148@tcp:/lustre 7.4G 40M 6.9G 1% /mnt/lustre2 PASS 35 (68s) == sanityn test 36: handle ESTALE/open-unlink correctly == 02:58:47 (1766390327) striped dir -i0 -c2 -H crush2 /mnt/lustre/d36.sanityn Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg148-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.58712 s, 89.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8196 Waiting for MDT destroys to complete *** cycle(0) *** before(7174040) after_dd(7122840) after(7174040) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.605367 s, 86.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8196 Waiting for MDT destroys to complete *** cycle(1) *** before(7174040) after_dd(7122840) after(7174040) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.672292 s, 78.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8196 Waiting for MDT destroys to complete *** cycle(2) *** before(7174040) after_dd(7122840) after(7174040) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.572013 s, 91.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8196 Waiting for MDT destroys to complete *** cycle(3) *** before(7174040) after_dd(7122840) after(7174040) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.651024 s, 80.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8196 Waiting for MDT destroys to complete *** cycle(4) *** before(7174040) after_dd(7122840) after(7174040) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.721654 s, 72.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8196 Waiting for MDT destroys to complete *** cycle(5) *** before(7174040) after_dd(7122840) after(7174040) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.675028 s, 77.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8196 Waiting for MDT destroys to complete *** cycle(6) *** before(7174040) after_dd(7122840) after(7174040) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.63619 s, 82.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8196 Waiting for MDT destroys to complete *** cycle(7) *** before(7174040) after_dd(7122840) after(7174040) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.61994 s, 84.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8196 Waiting for MDT destroys to complete *** cycle(8) *** before(7174040) after_dd(7122840) after(7174040) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.587399 s, 89.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8196 Waiting for MDT destroys to complete *** cycle(9) *** before(7174040) after_dd(7122840) after(7174040) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.572436 s, 91.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8196 Waiting for MDT destroys to complete *** cycle(10) *** before(7174040) after_dd(7122840) after(7174040) PASS 36 (197s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 03:02:04 (1766390524) striped dir -i1 -c2 -H all_char /mnt/lustre/d37.sanityn multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.8196 - create 2656 (time 1766390536.81 total 10.00 last 265.55) - create 5299 (time 1766390546.81 total 20.00 last 264.26) - create 9051 (time 1766390556.81 total 30.01 last 375.12) total: 10000 create in 33.02 seconds: 302.86 ops/second PASS 37 (38s) == sanityn test 39a: file mtime does not change after rename ========================================================== 03:02:42 (1766390562) repeat after cancel_lru_locks PASS 39a (3s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 03:02:45 (1766390565) repeat after cancel_lru_locks PASS 39b (4s) == sanityn test 39c: check truncate mtime update ================================================================================ 03:02:49 (1766390569) repeat after cancel_lru_locks PASS 39c (4s) == sanityn test 39d: sync write should update mtime ====== 03:02:53 (1766390573) fail_loc=0x411 fail_loc=0 PASS 39d (3s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 03:02:57 (1766390577) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (10s) == sanityn test 40c: pdirops: link and others ======================================================================== 03:03:07 (1766390587) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (10s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 03:03:17 (1766390597) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=8 No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (10s) == sanityn test 40e: pdirops: rename and others ======================================================================== 03:03:27 (1766390607) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (9s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 03:03:36 (1766390616) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=6 mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (6s) == sanityn test 41b: pdirops: create vs create ======================================================================== 03:03:42 (1766390622) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 open: File exists Conflict PASS 41b (7s) == sanityn test 41c: pdirops: create vs link ======================================================================== 03:03:49 (1766390629) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (8s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 03:03:57 (1766390637) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 41d (6s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 03:04:04 (1766390644) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 41e (7s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 03:04:11 (1766390651) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 41f (8s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 03:04:18 (1766390658) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=6 Conflict PASS 41g (7s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 03:04:25 (1766390665) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 Conflict PASS 41h (6s) == sanityn test 41i: reint_open: create vs create ======== 03:04:31 (1766390671) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (740s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 03:16:51 (1766391411) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (6s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 03:16:57 (1766391417) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=4 open: File exists Conflict PASS 42b (6s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 03:17:03 (1766391423) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (6s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 03:17:09 (1766391429) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 Conflict PASS 42d (6s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 03:17:15 (1766391435) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=1 mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (6s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 03:17:21 (1766391441) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=5 Conflict PASS 42f (6s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 03:17:27 (1766391447) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=4 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=1 Conflict PASS 42g (6s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 03:17:33 (1766391453) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 Conflict PASS 42h (6s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 03:17:39 (1766391459) PASS 43a (34s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 03:18:13 (1766391493) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=4 Conflict PASS 43b (6s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 03:18:19 (1766391499) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 Conflict PASS 43c (5s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 03:18:24 (1766391504) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=6 rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (6s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 03:18:30 (1766391510) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 43e (11s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 03:18:41 (1766391521) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (12s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 03:18:53 (1766391533) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (13s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 03:19:06 (1766391546) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 43h (15s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 03:19:21 (1766391561) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 43i (16s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 03:19:37 (1766391577) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (91s) == sanityn test 43k: unlink vs create ==================== 03:21:08 (1766391668) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (894s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 03:36:02 (1766392562) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Mon Dec 22 03:36:07 EST 2025 PASS 44a (8s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 03:36:10 (1766392570) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 open: File exists Conflict PASS 44b (9s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 03:36:19 (1766392579) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (8s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 03:36:27 (1766392587) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 44d (8s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 03:36:35 (1766392595) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 44e (8s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 03:36:43 (1766392603) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=8 Conflict PASS 44f (8s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 03:36:51 (1766392611) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 44g (7s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 03:36:58 (1766392618) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 44h (8s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 03:37:06 (1766392626) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 lfs mkdir: dirstripe error on '/mnt/lustre2/f44i.sanityn': stripe already set lfs setdirstripe: cannot create dir '/mnt/lustre2/f44i.sanityn': File exists Conflict PASS 44i (8s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 03:37:14 (1766392634) PASS 45a (91s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 03:38:45 (1766392725) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45b (8s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 03:38:53 (1766392733) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45c (8s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 03:39:01 (1766392741) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=6 rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (7s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 03:39:08 (1766392748) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45e (8s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 03:39:16 (1766392756) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (7s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 03:39:24 (1766392764) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=5 stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (7s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 03:39:31 (1766392771) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45h (8s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 03:39:39 (1766392779) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45i (8s) == sanityn test 45j: read vs rename ====================== 03:39:47 (1766392787) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (629s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 03:50:16 (1766393416) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (6s) == sanityn test 46b: pdirops: link vs create ======================================================================== 03:50:22 (1766393422) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=5 open: File exists Conflict PASS 46b (6s) == sanityn test 46c: pdirops: link vs link ======================================================================== 03:50:28 (1766393428) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=5 link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (6s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 03:50:34 (1766393434) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=5 Conflict PASS 46d (6s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 03:50:40 (1766393440) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 Conflict PASS 46e (6s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 03:50:46 (1766393446) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=8 Conflict PASS 46f (6s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 03:50:52 (1766393452) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 46g (7s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 03:50:59 (1766393459) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 46h (6s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 03:51:05 (1766393465) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 lfs mkdir: dirstripe error on '/mnt/lustre2/f46i.sanityn': stripe already set lfs setdirstripe: cannot create dir '/mnt/lustre2/f46i.sanityn': File exists Conflict PASS 46i (7s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 03:51:12 (1766393472) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mkdir: cannot create directory '/mnt/lustre2/f47a.sanityn': File exists Conflict PASS 47a (6s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 03:51:18 (1766393478) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=1 open: File exists Conflict PASS 47b (7s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 03:51:25 (1766393485) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f47c.sanityn' to '/mnt/lustre2/f47c.sanityn-2': File exists Conflict PASS 47c (6s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 03:51:31 (1766393491) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=1 Conflict PASS 47d (6s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 03:51:38 (1766393498) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=1 mv: cannot overwrite directory '/mnt/lustre2/f47e.sanityn' with non-directory Conflict PASS 47e (5s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 03:51:43 (1766393503) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=1 Conflict PASS 47f (6s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 03:51:49 (1766393509) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 47g (8s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 03:51:57 (1766393517) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.00450702 s, 2.3 MB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (7s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 03:52:04 (1766393524) 0+1 records in 0+1 records out 158 bytes copied, 0.0057933 s, 27.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3510: kill: (285081) - No such process PASS 51a (5s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 03:52:09 (1766393529) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00612379 s, 167 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244813 s, 42.8 MB/s 3145728 PASS 51b (15s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 03:52:24 (1766393544) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00680062 s, 151 kB/s PASS 51c (7s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 03:52:31 (1766393551) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213576 s, 49.1 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (4s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 03:52:35 (1766393555) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 1324 0x52c 0x280000401 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 1324 0x52c 0x280000401 read lease(1) has applied. PASS 51e (5s) == sanityn test 54: rename locking ======================= 03:52:40 (1766393560) llite.lustre-ffff9e1b82f6d000.xattr_cache=1 llite.lustre-ffff9e1b83169800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798762677 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 03:52:40.000000000 -0500 Modify: 2025-12-22 03:52:40.000000000 -0500 Change: 2025-12-22 03:52:40.000000000 -0500 Birth: 2025-12-22 03:52:41.000000000 -0500 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205289281278 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 03:52:46.000000000 -0500 Modify: 2025-12-22 03:52:46.000000000 -0500 Change: 2025-12-22 03:52:46.000000000 -0500 Birth: 2025-12-22 03:52:47.000000000 -0500 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798762680 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 03:52:52.000000000 -0500 Modify: 2025-12-22 03:52:52.000000000 -0500 Change: 2025-12-22 03:52:52.000000000 -0500 Birth: 2025-12-22 03:52:52.000000000 -0500 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205289281279 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 03:52:52.000000000 -0500 Modify: 2025-12-22 03:52:52.000000000 -0500 Change: 2025-12-22 03:52:52.000000000 -0500 Birth: 2025-12-22 03:52:52.000000000 -0500 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205289281281 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 03:52:57.000000000 -0500 Modify: 2025-12-22 03:52:58.000000000 -0500 Change: 2025-12-22 03:52:58.000000000 -0500 Birth: 2025-12-22 03:52:58.000000000 -0500 File: /mnt/lustre/d1 Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798762681 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 03:52:57.000000000 -0500 Modify: 2025-12-22 03:52:58.000000000 -0500 Change: 2025-12-22 03:52:58.000000000 -0500 Birth: 2025-12-22 03:52:58.000000000 -0500 PASS 54 (25s) == sanityn test 55a: rename vs unlink target dir ========= 03:53:05 (1766393585) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (8s) == sanityn test 55b: rename vs unlink source dir ========= 03:53:13 (1766393593) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (8s) == sanityn test 55c: rename vs unlink orphan target dir == 03:53:21 (1766393601) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (13s) == sanityn test 55d: rename file vs link ================= 03:53:34 (1766393614) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (10s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 03:53:44 (1766393624) fail_loc=0x155 We survived after AB/BA race test fail_loc=0x0 PASS 55e (13s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 03:53:58 (1766393638) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (14s) == sanityn test 55g: rename: race with trylock =========== 03:54:11 (1766393651) mdt.lustre-MDT0000.enable_rename_trylock=1 mdt.lustre-MDT0001.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (14s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 03:54:25 (1766393665) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1766393666 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 12.2078 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (8s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 03:54:33 (1766393673) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1766393674 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 4.20946 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (33s) == sanityn test 60: Verify data_version behaviour ======== 03:55:06 (1766393706) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60.sanityn PASS 60 (2s) == sanityn test 70a: cd directory && rm directory ======== 03:55:08 (1766393708) PASS 70a (3s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 03:55:11 (1766393711) PASS 70b (3s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 03:55:14 (1766393714) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00298474 s, 22.0 MB/s 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00285127 s, 23.0 MB/s File: /mnt/lustre2/f71a.sanityn Size: 262144 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205289281321 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 03:55:14.000000000 -0500 Modify: 2025-12-22 03:55:14.000000000 -0500 Change: 2025-12-22 03:55:14.000000000 -0500 Birth: 2025-12-22 03:55:15.000000000 -0500 26348 extent 0 in offset 65536, length 65536 flags (0x80000000): NET extent 1 in offset 196608, length 65536 flags (0x80000001): LAST NET No unwritten extents, extents number 2, sum of lengths 131072, expected sum 131072 26349 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00275132 s, 23.8 MB/s 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00343546 s, 19.1 MB/s File: /mnt/lustre/f71a.sanityn Size: 262144 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205289281321 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 03:55:14.000000000 -0500 Modify: 2025-12-22 03:55:14.000000000 -0500 Change: 2025-12-22 03:55:14.000000000 -0500 Birth: 2025-12-22 03:55:15.000000000 -0500 26350 extent 0 in offset 65536, length 65536 flags (0x80000000): NET extent 1 in offset 196608, length 65536 flags (0x80000001): LAST NET No unwritten extents, extents number 2, sum of lengths 131072, expected sum 131072 26349 PASS 71a (2s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 03:55:16 (1766393716) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00591945 s, 11.1 MB/s extent 0 in offset 0, length 65536 flags (0x80000001): LAST NET No unwritten extents, extents number 1, sum of lengths 65536, expected sum 65536 PASS 71b (2s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 03:55:18 (1766393718) filefrag 1.47.3-wc2 (11-Nov-2025) check last flag for file with 291 extents 290: 296960.. 296963: 378880.. 378883: 4: 0000: last,net,eof check last flag for file with 292 extents 291: 297984.. 297987: 494592.. 494595: 4: 0001: last,net,eof check last flag for file with 293 extents 292: 299008.. 299011: 708608.. 708611: 4: 0000: last,net,eof PASS 71c (12s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 03:55:30 (1766393730) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00542537 s, 12.1 MB/s 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00219597 s, 29.8 MB/s 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00188462 s, 34.8 MB/s 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.011857 s, 5.5 MB/s disk usage: 262144 /mnt/lustre/f71d.sanityn file size: 458752 /mnt/lustre/f71d.sanityn extent 0 in offset 0, length 65536 flags (0x80000000): NET extent 1 in offset 131072, length 65536 flags (0x80000000): NET extent 2 in offset 262144, length 65536 flags (0x80000000): NET extent 3 in offset 393216, length 65536 flags (0x80000001): LAST NET No unwritten extents, extents number 4, sum of lengths 262144, expected sum 262144 PASS 71d (15s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 03:55:45 (1766393745) llite.lustre-ffff9e1b82f6d000.xattr_cache=1 llite.lustre-ffff9e1b83169800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (2s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 03:55:47 (1766393747) llite.lustre-ffff9e1b82f6d000.xattr_cache=1 llite.lustre-ffff9e1b83169800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (3s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 03:55:50 (1766393750) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00439888 s, 233 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00362006 s, 283 kB/s 300594: get lock2 300596: get lock1 300596: done 300596: get lock2 300594: done 300594: get lock1 fcntl cmd 7 failed: Resource deadlock avoided 300596: cannot get lock2: Resource deadlock avoided 300596: exit rc=0 300594: done 300594: exit rc=0 PASS 74 (6s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 03:55:56 (1766393756) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0247151 s, 84.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152685 s, 68.7 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4193: 301185 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (7s) == sanityn test 76: Verify MDT open_files listing ======== 03:56:03 (1766393763) striped dir -i0 -c2 -H all_char /mnt/lustre/d76.sanityn opening files: ................ closing files: ................ PASS 76 (53s) == sanityn test 77a: check FIFO NRS policy =============== 03:56:56 (1766393816) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.165473 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438176 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718423 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583273 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688287 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547153 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921302 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100558 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090829 s, 11.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988984 s, 10.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e1b83169800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915407 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109268 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123542 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145309 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122093 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136536 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123975 s, 8.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e1b83169800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9e1b83169800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00911263 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120808 s, 86.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151167 s, 69.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147444 s, 71.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191687 s, 54.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176249 s, 59.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239647 s, 43.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01891 s, 55.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163467 s, 64.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170478 s, 61.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00546276 s, 192 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00962608 s, 109 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220476 s, 47.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0048688 s, 215 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00318746 s, 329 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00259208 s, 405 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg148-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg148-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (3s) == sanityn test 77b: check CRR-N NRS policy ============== 03:56:59 (1766393819) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.19303 s, 86.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431551 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523676 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643736 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855437 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083859 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853605 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983554 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100974 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113361 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110648 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891685 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10503 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102965 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103409 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100347 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769239 s, 13.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e1b83169800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9e1b83169800.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158857 s, 66.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131791 s, 79.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195677 s, 53.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225751 s, 46.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183114 s, 57.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185222 s, 56.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179466 s, 58.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276034 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255017 s, 41.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214561 s, 48.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216821 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268661 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232479 s, 45.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e1b83169800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00432328 s, 243 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00447787 s, 234 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00338816 s, 309 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg148-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg148-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.167517 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576645 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930118 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114983 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100679 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752626 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124382 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118074 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114516 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107524 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112132 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128087 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125489 s, 8.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153145 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13908 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139093 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14284 s, 7.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152322 s, 68.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00278305 s, 377 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202918 s, 51.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351512 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367554 s, 28.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210152 s, 49.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223379 s, 46.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172837 s, 60.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202012 s, 51.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149773 s, 70.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148727 s, 70.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182216 s, 57.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187842 s, 55.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181346 s, 57.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9e1b83169800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168631 s, 62.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00455884 s, 230 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg148-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg148-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (5s) == sanityn test 77c: check ORR NRS policy ================ 03:57:04 (1766393824) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.167292 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547829 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707972 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108253 s, 9.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9e1b83169800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139632 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12597 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132539 s, 7.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122218 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154571 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127281 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135657 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151178 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135743 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153666 s, 6.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144587 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140766 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136404 s, 7.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e1b83169800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9e1b83169800.lock_unused_count=4 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247773 s, 42.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150131 s, 69.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124628 s, 84.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123608 s, 84.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269515 s, 38.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146831 s, 71.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324235 s, 32.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299432 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318411 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283208 s, 37.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031053 s, 33.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274178 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400675 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264885 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272085 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270963 s, 38.7 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg148-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg148-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.235692 s, 71.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032373 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027314 s, 38.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520697 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821992 s, 12.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e1b83169800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948615 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112822 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122402 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13914 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15365 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14201 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159873 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130684 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139612 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146629 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126395 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00986825 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158201 s, 66.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14364 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218825 s, 47.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291266 s, 36.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129128 s, 81.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00865827 s, 121 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00315989 s, 332 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00424311 s, 247 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00725981 s, 144 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00276541 s, 379 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00835949 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00942018 s, 111 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00349298 s, 300 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00476364 s, 220 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0070514 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00724387 s, 145 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e1b83169800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9e1b83169800.lock_unused_count=2 removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg148-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg148-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.169779 s, 98.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432501 s, 24.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359534 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939012 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103956 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129199 s, 8.1 MB/sldlm.namespaces.lustre-OST0000-osc-ffff9e1b83169800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9e1b83169800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127571 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125179 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145738 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144711 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134053 s, 7.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158474 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146218 s, 7.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138842 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167913 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163599 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151056 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264249 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223426 s, 46.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00689143 s, 152 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259466 s, 40.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334201 s, 31.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299205 s, 35.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325198 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280487 s, 37.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432173 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257966 s, 40.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333105 s, 31.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302275 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034771 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340308 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036779 s, 28.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e1b83169800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9e1b83169800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357948 s, 29.3 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg148-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg148-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (6s) == sanityn test 77d: check TRR nrs policy ================ 03:57:10 (1766393830) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.17866 s, 93.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434329 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446931 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695477 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481989 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081264 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982774 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115667 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106599 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123449 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113535 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136335 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121084 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134617 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130143 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133129 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135094 s, 7.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102759 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010293 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109274 s, 96.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010953 s, 95.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124586 s, 84.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00656845 s, 160 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e1b83169800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9e1b83169800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307417 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363273 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317283 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035533 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041707 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00409385 s, 256 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0041125 s, 255 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00445884 s, 235 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281529 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00291381 s, 360 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg148-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg148-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.240606 s, 69.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596864 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472214 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632896 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979908 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105397 s, 99.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172013 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177055 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168669 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331528 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382977 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188742 s, 55.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146136 s, 71.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318099 s, 33.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121316 s, 86.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166327 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397579 s, 26.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445928 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173566 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179438 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472721 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114712 s, 91.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181894 s, 5.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416868 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17977 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176481 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16699 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0092228 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109326 s, 95.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193411 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0094918 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294853 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205673 s, 5.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e1b83169800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9e1b83169800.lock_unused_count=3 removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg148-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg148-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.171917 s, 97.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278791 s, 37.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320585 s, 32.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509492 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250227 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830258 s, 12.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e1b83169800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107019 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116195 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1345 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11719 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135334 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104482 s, 10.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134122 s, 7.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145187 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149269 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154465 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120928 s, 8.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187881 s, 55.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221529 s, 47.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023393 s, 44.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231629 s, 45.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241987 s, 43.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00492769 s, 213 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278166 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228125 s, 46.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305426 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337086 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282976 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286714 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431862 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351587 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00687745 s, 152 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330938 s, 31.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e1b83169800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9e1b83169800.lock_unused_count=6 removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg148-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg148-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (5s) == sanityn test 77e: check TBF NID nrs policy ============ 03:57:15 (1766393835) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.201.48@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.167759 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370945 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642555 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602958 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564609 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778787 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792334 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617642 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131756 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107535 s, 9.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9e1b83169800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113206 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125683 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122299 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130665 s, 8.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139801 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145571 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13244 s, 7.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020167 s, 52.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333822 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361513 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295981 s, 35.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277052 s, 37.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225093 s, 46.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225544 s, 46.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032139 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279111 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286994 s, 36.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336156 s, 31.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335928 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317662 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298428 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294008 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344186 s, 30.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e1b83169800.lock_unused_count=2 removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg148-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg148-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.146901 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302755 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039184 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347226 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133007 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108535 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118543 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135816 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126202 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141988 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137229 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149399 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143973 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124888 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12538 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131541 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131026 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120777 s, 86.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188664 s, 55.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124155 s, 84.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235482 s, 44.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175432 s, 59.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241326 s, 43.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145319 s, 72.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198046 s, 52.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127175 s, 82.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172064 s, 60.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00653179 s, 161 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244736 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250576 s, 41.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00485764 s, 216 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0052318 s, 200 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149594 s, 70.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e1b83169800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9e1b83169800.lock_unused_count=3 removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg148-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg148-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.153186 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378636 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307894 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135244 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123832 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127095 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160773 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146971 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145767 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144787 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144135 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159769 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156329 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16613 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151292 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155916 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14595 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026934 s, 38.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298745 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245993 s, 42.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220456 s, 47.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151644 s, 69.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026867 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266385 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248423 s, 42.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258853 s, 40.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329549 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280257 s, 37.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350743 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324034 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294746 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312924 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311886 s, 33.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e1b83169800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9e1b83169800.lock_unused_count=2 removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg148-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg148-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.143407 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305848 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371708 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982507 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101672 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911709 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884429 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841053 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107126 s, 9.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e1b83169800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9e1b83169800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874601 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11397 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125698 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128391 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112962 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127888 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112157 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112704 s, 9.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155458 s, 67.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134996 s, 77.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215064 s, 48.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167735 s, 62.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241863 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015789 s, 66.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188996 s, 55.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267954 s, 39.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185542 s, 56.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286738 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020099 s, 52.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033902 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237056 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249295 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290965 s, 36.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e1b83169800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9e1b83169800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319892 s, 32.8 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg148-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg148-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (8s) == sanityn test 77f: check TBF JobID nrs policy ========== 03:57:23 (1766393843) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg148-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.233088 s, 72.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361089 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367412 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173537 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174267 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165772 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171082 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182887 s, 5.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202153 s, 5.2 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176082 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175336 s, 6.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162832 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205309 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183353 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194333 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176668 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209034 s, 5.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196756 s, 53.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0037105 s, 283 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022894 s, 45.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226493 s, 46.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135011 s, 77.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117309 s, 89.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148526 s, 70.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146313 s, 71.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00312101 s, 336 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010607 s, 98.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159329 s, 65.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013336 s, 78.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251015 s, 41.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00385643 s, 272 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0033453 s, 313 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e1b83169800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9e1b83169800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00422434 s, 248 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg148-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg148-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg148-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.180772 s, 92.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517913 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890793 s, 11.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939044 s, 11.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113105 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135239 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921685 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932173 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142446 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11148 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140885 s, 7.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 5001+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126764 s, 8.3 MB/s [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170288 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147431 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150001 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134532 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17206 s, 6.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e1b83169800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9e1b83169800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101793 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131478 s, 79.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206401 s, 50.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00384357 s, 273 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00289174 s, 363 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0039689 s, 264 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00409308 s, 256 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00333194 s, 315 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00403241 s, 260 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00417177 s, 251 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024427 s, 42.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0041577 s, 252 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278222 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.005893 s, 178 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00448261 s, 234 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00438501 s, 239 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg148-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg148-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg148-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.184868 s, 90.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308468 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419181 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057199 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833364 s, 12.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9e1b83169800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08427 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753575 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127442 s, 8.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137924 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143554 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125465 s, 8.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129715 s, 8.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134324 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142345 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131154 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145878 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149774 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237422 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223564 s, 46.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288612 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260097 s, 40.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258711 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355433 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334677 s, 31.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329646 s, 31.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409098 s, 25.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348058 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329771 s, 31.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342261 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351045 s, 29.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390787 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392292 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393289 s, 26.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e1b83169800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9e1b83169800.lock_unused_count=3 removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg148-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg148-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg148-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.181911 s, 92.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103216 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109323 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114019 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107434 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120032 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105526 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117095 s, 9.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e1b83169800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111173 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127345 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11642 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112407 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109414 s, 9.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955084 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127132 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120383 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108099 s, 9.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173502 s, 60.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146862 s, 71.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0039078 s, 268 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9e1b83169800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261459 s, 40.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411933 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00306335 s, 342 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315837 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393315 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00324592 s, 323 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201531 s, 52.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365974 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168735 s, 62.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155303 s, 67.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144077 s, 72.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111303 s, 94.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00976007 s, 107 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg148-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg148-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (9s) == sanityn test 77g: Change TBF type directly ============ 03:57:32 (1766393852) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (4s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 03:57:36 (1766393856) oleg148-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg148-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg148-client: oleg148-server: ssh exited with exit code 2 oleg148-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg148-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg148-client: oleg148-server: ssh exited with exit code 22 oleg148-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg148-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg148-client: oleg148-server: ssh exited with exit code 22 oleg148-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg148-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg148-client: oleg148-server: ssh exited with exit code 22 PASS 77h (4s) == sanityn test 77i: Change rank of TBF rule ============= 03:57:40 (1766393860) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (8s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 03:57:48 (1766393868) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.162082 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377401 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038839 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139376 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141611 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137029 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156768 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136195 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14528 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139135 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128626 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160478 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151465 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140785 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160582 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165877 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145456 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026517 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331386 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224832 s, 46.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233184 s, 45.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255551 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200255 s, 52.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288277 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355338 s, 29.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382189 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351055 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298262 s, 35.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302377 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221359 s, 47.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042849 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337243 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320738 s, 32.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e1b83169800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9e1b83169800.lock_unused_count=2 removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg148-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg148-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.90178 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4414 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (42s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 03:58:30 (1766393910) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg148-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg148-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg148-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg148-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg148-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.149131 s, 112 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067181 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111904 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109203 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118936 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108389 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10909 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938903 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120678 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126766 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114031 s, 9.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130043 s, 8.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123684 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145254 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121676 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128323 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120625 s, 8.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244689 s, 42.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024369 s, 43.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262819 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220166 s, 47.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224141 s, 46.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273903 s, 38.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205446 s, 51.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213809 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309483 s, 33.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317169 s, 33.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180096 s, 58.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242632 s, 43.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242838 s, 43.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273201 s, 38.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241654 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268593 s, 39.0 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg148-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg148-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6784 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9789 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg148-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg148-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg148-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg148-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg148-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.286014 s, 58.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426871 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512143 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061952 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752938 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745413 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796467 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932261 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117767 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114984 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133295 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13081 s, 8.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145443 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196238 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200876 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201198 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186951 s, 5.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219051 s, 47.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242593 s, 43.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273397 s, 38.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238144 s, 44.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287926 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304575 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236866 s, 44.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306422 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250007 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287248 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366302 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361053 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381366 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333476 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331705 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388665 s, 27.0 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg148-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg148-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6843 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9636 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (112s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 04:00:22 (1766394022) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg148-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg148-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg148-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg148-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg148-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.286829 s, 58.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722137 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681319 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960272 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818343 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070298 s, 14.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0928588 s, 11.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107126 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121427 s, 8.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109953 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120958 s, 8.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123808 s, 8.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110737 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131063 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128276 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140736 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158785 s, 6.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016432 s, 63.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188026 s, 55.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173923 s, 60.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244608 s, 42.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024891 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311298 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218633 s, 48.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331335 s, 31.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341278 s, 30.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243481 s, 43.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247016 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233957 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293381 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235264 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0081579 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303309 s, 34.6 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg148-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg148-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7022 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9576 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg148-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg148-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg148-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg148-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg148-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.27191 s, 61.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603659 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150462 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146261 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177726 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149846 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147603 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172403 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189975 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169325 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173241 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16902 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175528 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168509 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171213 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166895 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225814 s, 4.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199487 s, 52.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194825 s, 53.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223508 s, 46.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165072 s, 63.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286061 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023952 s, 43.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236929 s, 44.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259178 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029837 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311971 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225957 s, 46.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026035 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021237 s, 49.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348275 s, 30.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317911 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299041 s, 35.1 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg148-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg148-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6581 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.955 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (113s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 04:02:15 (1766394135) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg148-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.146378 s, 115 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577678 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727496 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915496 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10102 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966755 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130946 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12683 s, 8.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9e1b83169800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1193 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127103 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127154 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131653 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126179 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123019 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146128 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127816 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1496 s, 7.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165101 s, 63.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198678 s, 52.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204138 s, 51.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127426 s, 82.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171129 s, 61.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167878 s, 62.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016617 s, 63.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205232 s, 51.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00346652 s, 302 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254858 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222057 s, 47.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00410356 s, 256 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193792 s, 54.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164335 s, 63.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00373688 s, 281 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140267 s, 74.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e1b83169800.lock_unused_count=2 removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg148-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg148-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.90826 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74584 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.201.48@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.201.48@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.145836 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290755 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512544 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513631 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620365 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663575 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609703 s, 17.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9e1b83169800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968059 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101984 s, 10.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153838 s, 68.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182549 s, 57.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022923 s, 45.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192105 s, 54.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125014 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00305327 s, 343 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115115 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131618 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00894808 s, 117 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109938 s, 95.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405632 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117642 s, 89.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179819 s, 58.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375152 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143747 s, 7.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142338 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112083 s, 9.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214467 s, 48.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140877 s, 7.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252038 s, 41.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240714 s, 43.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134824 s, 77.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153413 s, 68.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156595 s, 6.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e1b83169800.lock_unused_count=9 ldlm.namespaces.lustre-OST0001-osc-ffff9e1b83169800.lock_unused_count=2 removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg148-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg148-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.89853 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73556 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.201.48@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg148-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.164302 s, 102 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316348 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622952 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642747 s, 16.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e1b83169800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9e1b83169800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185283 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187757 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17793 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153441 s, 6.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187064 s, 5.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179729 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179811 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163681 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178337 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179308 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185991 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182588 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179482 s, 5.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270863 s, 38.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304721 s, 34.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132438 s, 79.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108123 s, 97.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137429 s, 76.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252577 s, 41.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286912 s, 36.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182209 s, 57.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269984 s, 38.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190776 s, 55.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322412 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00969403 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228304 s, 45.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120071 s, 87.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260486 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00893865 s, 117 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e1b83169800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9e1b83169800.lock_unused_count=1 removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg148-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg148-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.91155 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.88302 s, 21.5 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg148-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.153166 s, 110 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376188 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437605 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310497 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780364 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982442 s, 10.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9e1b83169800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102207 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125624 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12008 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127592 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139513 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127441 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124408 s, 8.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e1b83169800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9e1b83169800.lock_unused_count=3 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130199 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140839 s, 7.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139856 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125194 s, 8.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147286 s, 71.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141253 s, 74.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195495 s, 53.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145492 s, 72.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146725 s, 71.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170324 s, 61.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176334 s, 59.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00526804 s, 199 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190701 s, 55.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212489 s, 49.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206312 s, 50.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204596 s, 51.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223355 s, 46.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00510405 s, 205 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00544812 s, 192 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236397 s, 44.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg148-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg148-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74949 s, 10.8 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73434 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.89729 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73977 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg148-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.153696 s, 109 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266714 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525574 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675908 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457513 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829276 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807555 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087505 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934816 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111268 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122557 s, 8.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115964 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101675 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12071 s, 8.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124463 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112607 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101808 s, 10.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157938 s, 66.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114652 s, 91.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131702 s, 79.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147641 s, 71.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157565 s, 66.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193628 s, 54.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0079121 s, 133 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222605 s, 47.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235725 s, 44.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00570327 s, 184 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243476 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00308791 s, 340 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215585 s, 48.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00874267 s, 120 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00920914 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00782398 s, 134 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg148-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg148-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6475 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.432 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg148-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.15965 s, 105 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752735 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818584 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103853 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104325 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881252 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999305 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1308 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113862 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132759 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156165 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16222 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171924 s, 6.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e1b83169800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9e1b83169800.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195902 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177982 s, 5.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143499 s, 73.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241604 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227598 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187428 s, 55.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306564 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00583877 s, 180 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027918 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00829872 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153873 s, 68.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231828 s, 4.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311762 s, 33.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290236 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00500431 s, 210 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317708 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0072841 s, 144 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162433 s, 64.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015109 s, 69.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231701 s, 45.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg148-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg148-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.90181 s, 21.4 MB/s Write runtime is 5 s, speed is 20.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73695 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg148-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.190388 s, 88.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079394 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121282 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140072 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134512 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127459 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138664 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124893 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160534 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148433 s, 7.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9e1b83169800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221035 s, 4.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2151 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220673 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230178 s, 4.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235316 s, 4.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268301 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286558 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273456 s, 38.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229062 s, 45.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442814 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281182 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03053 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401519 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322598 s, 32.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385573 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397472 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403668 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419891 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416695 s, 25.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482035 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444877 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653879 s, 16.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e1b83169800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480697 s, 21.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg148-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg148-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.75866 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73629 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74623 s, 10.8 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73704 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (272s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 04:06:47 (1766394407) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg148-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.188073 s, 89.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420663 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711283 s, 14.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9e1b83169800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16154 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152073 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169866 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163776 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19489 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220063 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209138 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215633 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209457 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209975 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193102 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203333 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199421 s, 5.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0000-osc-ffff9e1b83169800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9e1b83169800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20249 s, 5.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237541 s, 44.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136707 s, 76.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161435 s, 65.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016129 s, 65.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169512 s, 61.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02287 s, 45.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194524 s, 53.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255557 s, 41.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224554 s, 46.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00756821 s, 139 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107487 s, 97.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302827 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00530993 s, 197 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173841 s, 60.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00495214 s, 212 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00731947 s, 143 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg148-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg148-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.91987 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7418 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn Waiting for MDT destroys to complete PASS 77kb (27s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 04:07:14 (1766394434) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.201.48@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.201.48@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.179754 s, 93.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876566 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102436 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902182 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10311 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938775 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970505 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092338 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987766 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10517 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118889 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114821 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111508 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101708 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117758 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116944 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136373 s, 7.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e1b83169800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9e1b83169800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971481 s, 10.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916649 s, 11.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109152 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101624 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958812 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899295 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915274 s, 11.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912201 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915776 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920553 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937052 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942552 s, 11.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954174 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101777 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964914 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100425 s, 10.4 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg148-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg148-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93017 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74433 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn Waiting for MDT destroys to complete PASS 77kc (29s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 04:07:43 (1766394463) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.201.48@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg148-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.178696 s, 93.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582653 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726157 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684016 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758477 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102399 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909935 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768541 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965175 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174644 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183165 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162781 s, 6.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162784 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184746 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187792 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183449 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227002 s, 4.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149716 s, 70.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024328 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247257 s, 42.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153349 s, 68.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361326 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293413 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330154 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275832 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311313 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279799 s, 37.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e1b83169800.lock_unused_count=8 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314441 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335462 s, 31.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032012 s, 32.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309277 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0070445 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00617355 s, 170 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg148-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg148-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9219 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.98487 s, 21.0 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn Waiting for MDT destroys to complete PASS 77kd (27s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 04:08:10 (1766394490) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg148-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.181094 s, 92.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464328 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460495 s, 22.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e1b83169800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9e1b83169800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146796 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156863 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16693 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178106 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188557 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203008 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174312 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215488 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2093 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224317 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200774 s, 5.2 MB/s 1+0 records in 1+0 records out running as uid/gid/euid/egid 500/500/500/500, groups: 5001+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199666 s, 5.3 MB/s [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212036 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233516 s, 4.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117839 s, 89.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186792 s, 56.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105969 s, 99.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297909 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276692 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280577 s, 37.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021301 s, 49.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283747 s, 37.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356335 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292362 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451289 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292979 s, 35.8 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff9e1b83169800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9e1b83169800.lock_unused_count=2 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427917 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425251 s, 24.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366393 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349797 s, 30.0 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg148-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg148-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.75734 s, 10.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73675 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.89835 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74794 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn Waiting for MDT destroys to complete PASS 77ke (55s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 04:09:05 (1766394545) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg148-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.155299 s, 108 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306254 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042022 s, 25.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610065 s, 17.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650838 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664212 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684411 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691532 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919353 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865494 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978917 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924415 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105521 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10853 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103478 s, 10.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120151 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11175 s, 9.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134953 s, 77.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174058 s, 60.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182532 s, 57.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158467 s, 66.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142091 s, 73.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231364 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116203 s, 90.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155855 s, 67.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274915 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022048 s, 47.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128699 s, 81.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00983178 s, 107 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117613 s, 89.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181668 s, 57.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00908107 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150549 s, 69.7 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg148-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg148-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4587 s, 5.4 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9664 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn Waiting for MDT destroys to complete PASS 77kf (51s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 04:09:56 (1766394596) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg148-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.173109 s, 96.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404981 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821559 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105018 s, 10.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e1b83169800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106037 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942332 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897696 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936309 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110809 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109386 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106612 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136753 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129462 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106129 s, 9.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274806 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327743 s, 32.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283166 s, 37.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242135 s, 43.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031772 s, 33.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221542 s, 47.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210236 s, 49.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321653 s, 32.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171178 s, 61.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181975 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389646 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175586 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244836 s, 42.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157578 s, 66.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299012 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0093787 s, 112 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9e1b83169800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184284 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147456 s, 71.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00408571 s, 257 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg148-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg148-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.90966 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73682 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg148-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.137655 s, 122 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272125 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380096 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327414 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100415 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087313 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133619 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150161 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146366 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132003 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144493 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153176 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129338 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139275 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142853 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138118 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134152 s, 7.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e1b83169800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9e1b83169800.lock_unused_count=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188997 s, 55.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214327 s, 48.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152479 s, 68.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336392 s, 31.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185247 s, 56.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00574642 s, 182 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279361 s, 37.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207099 s, 50.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222944 s, 47.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226989 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022617 s, 46.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231959 s, 45.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236361 s, 44.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254657 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268802 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275709 s, 38.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9e1b83169800.lock_unused_count=2 removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg148-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg148-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.75762 s, 10.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73763 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74922 s, 10.8 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73779 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn Waiting for MDT destroys to complete PASS 77kg (84s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 04:11:21 (1766394681) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (4s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 04:11:24 (1766394684) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg148-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.4686 s, 302 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg148-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0089 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (51s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 04:12:16 (1766394736) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.157485 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387085 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712605 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832683 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779714 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101915 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104264 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863502 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101568 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100189 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104257 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123548 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114478 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103235 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110234 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134398 s, 7.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e1b83169800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9e1b83169800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156365 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00303759 s, 345 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00550672 s, 190 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00584572 s, 179 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00506376 s, 207 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00472508 s, 222 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344712 s, 30.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282508 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374616 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372258 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307626 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381565 s, 27.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00415217 s, 253 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332338 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00441217 s, 238 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228023 s, 46.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00371706 s, 282 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg148-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg148-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.904 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97307 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.149551 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803309 s, 13.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899725 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785754 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799601 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103334 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154908 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161517 s, 6.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e1b83169800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174034 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178093 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171389 s, 6.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193105 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252334 s, 41.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155875 s, 67.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181119 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190121 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206083 s, 5.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221419 s, 4.7 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216148 s, 48.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026366 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204929 s, 51.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015487 s, 67.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255511 s, 41.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263685 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104909 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00980777 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342723 s, 30.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124395 s, 84.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191076 s, 54.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181458 s, 57.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00980105 s, 107 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202717 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00626226 s, 167 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg148-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg148-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.89854 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97115 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (40s) == sanityn test 77o: Changing rank should not panic ====== 04:12:56 (1766394776) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (4s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 04:13:00 (1766394780) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (39s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 04:13:39 (1766394819) mds.MDS.mdt.nrs_policies=tbf oleg148-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg148-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg148-client: oleg148-server: ssh exited with exit code 36 oleg148-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg148-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg148-client: oleg148-server: ssh exited with exit code 22 oleg148-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg148-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg148-client: oleg148-server: ssh exited with exit code 22 oleg148-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg148-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg148-client: oleg148-server: ssh exited with exit code 22 oleg148-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg148-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg148-client: oleg148-server: ssh exited with exit code 22 oleg148-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg148-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg148-client: oleg148-server: ssh exited with exit code 22 oleg148-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg148-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg148-client: oleg148-server: ssh exited with exit code 22 oleg148-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg148-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg148-client: oleg148-server: ssh exited with exit code 22 oleg148-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg148-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg148-client: oleg148-server: ssh exited with exit code 22 oleg148-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg148-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg148-client: oleg148-server: ssh exited with exit code 22 oleg148-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg148-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg148-client: oleg148-server: ssh exited with exit code 22 oleg148-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg148-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg148-client: oleg148-server: ssh exited with exit code 22 oleg148-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg148-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg148-client: oleg148-server: ssh exited with exit code 22 oleg148-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg148-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg148-client: oleg148-server: ssh exited with exit code 22 oleg148-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg148-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg148-client: oleg148-server: ssh exited with exit code 22 oleg148-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg148-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg148-client: oleg148-server: ssh exited with exit code 22 oleg148-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg148-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg148-client: oleg148-server: ssh exited with exit code 22 oleg148-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg148-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg148-client: oleg148-server: ssh exited with exit code 22 oleg148-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg148-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg148-client: oleg148-server: ssh exited with exit code 22 oleg148-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg148-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg148-client: oleg148-server: ssh exited with exit code 22 oleg148-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg148-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg148-client: oleg148-server: ssh exited with exit code 22 oleg148-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg148-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg148-client: oleg148-server: ssh exited with exit code 22 oleg148-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg148-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg148-client: oleg148-server: ssh exited with exit code 22 oleg148-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg148-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg148-client: oleg148-server: ssh exited with exit code 22 oleg148-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg148-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg148-client: oleg148-server: ssh exited with exit code 22 oleg148-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg148-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg148-client: oleg148-server: ssh exited with exit code 22 oleg148-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg148-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg148-client: oleg148-server: ssh exited with exit code 22 oleg148-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg148-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg148-client: oleg148-server: ssh exited with exit code 22 oleg148-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg148-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg148-client: oleg148-server: ssh exited with exit code 22 oleg148-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg148-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg148-client: oleg148-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (12s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 04:13:51 (1766394831) striped dir -i0 -c1 -H crush /mnt/lustre/d77r.sanityn mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 339258 kill md_thread xargs PASS 77r (42s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 04:14:33 (1766394873) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (4s) == sanityn test 79: xattr: intent error ================== 04:14:37 (1766394877) striped dir -i1 -c2 -H all_char /mnt/lustre/d79.sanityn File: /mnt/lustre/d79.sanityn Size: 8192 Blocks: 16 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798763917 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 04:14:37.000000000 -0500 Modify: 2025-12-22 04:14:37.000000000 -0500 Change: 2025-12-22 04:14:37.000000000 -0500 Birth: 2025-12-22 04:14:38.000000000 -0500 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (12s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 04:14:49 (1766394889) total: 10 open/close in 0.04 seconds: 265.76 ops/second multiop /mnt/lustre2/d80a.sanityn/f80a.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8196 Finish migration, then checking.. PASS 80a (4s) == sanityn test 80b: Accessing directory during migration ========================================================== 04:14:53 (1766394893) start migration thread 348464 accessing the migrating directory for 1 minute... concurrent access 1616 failures, 2344 successes /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 348464 Killed ( while true; do mdt_idx=$((RANDOM % MDSCOUNT)); $LFS migrate -m $mdt_idx $migdir1 &> /dev/null; done ) (wd: ~) PASS 80b (165s) == sanityn test 81a: rename and stat under striped directory ========================================================== 04:17:38 (1766395058) stat: cannot statx 'd0/0': No such file or directory File: d0/0 Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934277 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 04:17:39.000000000 -0500 Modify: 2025-12-22 04:17:39.000000000 -0500 Change: 2025-12-22 04:17:39.000000000 -0500 Birth: 2025-12-22 04:17:39.000000000 -0500 PASS 81a (2s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 04:17:40 (1766395060) total: 2500 open/close in 6.82 seconds: 366.80 ops/second /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 354368 Killed ( while true; do touch $DIR1/$tdir; done ) (wd: ~) PASS 81b (52s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 04:18:32 (1766395112) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (1s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 04:18:33 (1766395113) total: 2500 open/close in 5.99 seconds: 417.62 ops/second total: 2500 open/close in 5.93 seconds: 421.92 ops/second snapshot_time 1766395159.692788777 secs.nsecs start_time 1766395115.329713454 secs.nsecs elapsed_time 44.363075323 secs.nsecs open 2505 samples [usecs] 104 4333 826477 323192667 close 2504 samples [usecs] 46 2910 208142 27187470 mknod 2501 samples [usecs] 138 4228 698285 239759765 unlink 2500 samples [usecs] 207 64615 2019301 11628199949 mkdir 2 samples [usecs] 1022 2146 3168 5649800 rmdir 2 samples [usecs] 2571 3855 6426 21471066 rename 2500 samples [usecs] 199 14724 3637084 8228078386 getattr 3765 samples [usecs] 1 4690 139632 47769148 statfs 8 samples [usecs] 7 203 335 44057 parallel_rename_file 2500 samples [usecs] 199 14724 3637084 8228078386 crossdir_rename 2500 samples [usecs] 199 14724 3637084 8228078386 snapshot_time 1766395159.692895217 secs.nsecs start_time 1766395115.329771805 secs.nsecs elapsed_time 44.363123412 secs.nsecs open 2504 samples [usecs] 96 2758 911521 382584085 close 2504 samples [usecs] 44 2969 234103 35026433 mknod 2500 samples [usecs] 142 2683 776577 281393801 unlink 2500 samples [usecs] 201 38201 1884648 6159521538 mkdir 2 samples [usecs] 1025 6123 7148 38541754 rmdir 2 samples [usecs] 4279 5922 10201 53379925 rename 2500 samples [usecs] 208 8420 3034330 5322386630 getattr 3757 samples [usecs] 8 19084 182585 729143559 statfs 8 samples [usecs] 9 27 127 2331 parallel_rename_file 2500 samples [usecs] 208 8420 3034330 5322386630 crossdir_rename 2500 samples [usecs] 208 8420 3034330 5322386630 PASS 81d (48s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 04:19:21 (1766395161) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.8196 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.8196 PASS 82 (3s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 04:19:24 (1766395164) start pid 368794 to create/unlink striped directory start pid 368795 to stat striped directory /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5797: 368794 Terminated ( cd $DIR1; while true; do $LFS mkdir -i1 -c2 $tdir; rmdir $tdir; done ) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5797: 368795 Terminated ( cd $DIR2; while true; do stat $tdir > /dev/null 2>&1; done ) PASS 83 (122s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 04:21:26 (1766395286) fail_loc=0x8000060b PASS 84 (8s) == sanityn test 85: Lustre API root cache race =========== 04:21:34 (1766395294) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1766395294 Finishing test test1 at 1766395295 PASS 85 (2s) == sanityn test 90: open/create and unlink striped directory ========================================================== 04:21:37 (1766395297) start pid 379275 to open/create under striped directory start pid 379276 to unlink striped directory /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5864: 379275 Terminated ( cd $DIR1; while true; do $LFS mkdir -c$MDSCOUNT $tdir > /dev/null 2>&1; touch $tdir/f{0..3} > /dev/null 2>&1; done ) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5864: 379276 Terminated ( cd $DIR2; while true; do rm -rf $tdir > /dev/null 2>&1; done ) PASS 90 (182s) == sanityn test 91: chmod and unlink striped directory === 04:24:39 (1766395479) start pid 391119 to chmod striped directory start pid 391120 to unlink striped directory /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5901: 391119 Terminated ( cd $DIR1; while true; do $LFS mkdir -c$MDSCOUNT $tdir > /dev/null 2>&1; chmod go+w $tdir > /dev/null 2>&1; done ) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5901: 391120 Terminated ( cd $DIR2; while true; do rm -rf $tdir > /dev/null 2>&1; done ) PASS 91 (182s) == sanityn test 92: create remote directory under orphan directory ========================================================== 04:27:41 (1766395661) fail_loc=0x1408 mkdir: cannot create directory '/mnt/lustre2/d92.sanityn/dir': No such file or directory lfs setdirstripe: cannot resolve path '/mnt/lustre2/d92.sanityn/remote_dir': No such file or directory (2) lfs setdirstripe: '/mnt/lustre2/d92.sanityn/remote_dir' is not on a Lustre filesystem: No such file or directory (2) lfs setdirstripe: cannot create dir '/mnt/lustre2/d92.sanityn/remote_dir': No such file or directory fail_loc=0 PASS 92 (3s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 04:27:44 (1766395664) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5257 0x1489 0x2c0000400 0 5269 0x1495 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6475 0x194b 0x280000401 1 6429 0x191d 0x2c0000401 rc1=0 and rc2=0 PASS 93 (9s) == sanityn test 94: signal vs CP callback race =========== 04:27:53 (1766395673) osc.lustre-OST0000-osc-ffff9e1b82f6d000.idle_timeout=0 osc.lustre-OST0000-osc-ffff9e1b83169800.idle_timeout=0 osc.lustre-OST0001-osc-ffff9e1b82f6d000.idle_timeout=0 osc.lustre-OST0001-osc-ffff9e1b83169800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0145311 s, 564 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.09708 s, 1.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5988: 410870 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00473197 s, 866 kB/s osc.lustre-OST0000-osc-ffff9e1b82f6d000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9e1b83169800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e1b82f6d000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e1b83169800.idle_timeout=debug PASS 94 (10s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 04:28:03 (1766395683) llite.lustre-ffff9e1b82f6d000.fast_read=0 llite.lustre-ffff9e1b83169800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0201419 s, 104 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff9e1b82f6d000.fast_read=1 llite.lustre-ffff9e1b83169800.fast_read=1 PASS 95a (15s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 04:28:18 (1766395698) llite.lustre-ffff9e1b82f6d000.fast_read=0 llite.lustre-ffff9e1b83169800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.0018223 s, 6.7 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff9e1b82f6d000.fast_read=1 llite.lustre-ffff9e1b83169800.fast_read=1 PASS 95b (8s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 04:28:26 (1766395706) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (1s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 04:28:28 (1766395708) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00195564 s, 2.1 MB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (2s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 04:28:30 (1766395710) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0251588 s, 83.4 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (2s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 04:28:32 (1766395712) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0218041 s, 96.2 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (3s) == sanityn test 100e: DoM: read on open and file size ==== 04:28:35 (1766395715) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00170587 s, 7.0 MB/s PASS 100e (2s) == sanityn test 101a: Discard DoM data on unlink ========= 04:28:37 (1766395717) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000576177 s, 7.1 MB/s PASS 101a (2s) == sanityn test 101b: Discard DoM data on rename ========= 04:28:39 (1766395719) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00062052 s, 6.6 MB/s PASS 101b (2s) == sanityn test 101c: Discard DoM data on close-unlink === 04:28:41 (1766395721) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000714003 s, 5.7 MB/s PASS 101c (3s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 04:28:44 (1766395724) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 3c 1f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0x1f3c:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102.sanityn.remote fh_bytes: 32 fh_type: 151 fh_data: 02 04 00 40 02 00 00 00 b3 3a 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x240000402:0x3ab3:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102.sanityn.remote/subdir fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 3d 1f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0x1f3d:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102.sanityn.stripe fh_bytes: 32 fh_type: 151 fh_data: 02 04 00 40 02 00 00 00 b4 3a 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x240000402:0x3ab4:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102.sanityn.stripe/subdir fh_bytes: 32 fh_type: 151 fh_data: 02 04 00 40 02 00 00 00 b5 3a 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x240000402:0x3ab5:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! PASS 102 (3s) == sanityn test 103: Test size correctness with lockahead ========================================================== 04:28:47 (1766395727) striped dir -i1 -c2 -H all_char /mnt/lustre/d103.sanityn fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1766395728 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1766395730 Finishing test test23 at 1766395730 PASS 103 (7s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 04:28:54 (1766395734) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00267743 s, 382 kB/s oleg148-server: debugfs 1.47.3-wc2 (11-Nov-2025) STAT a|m|ctime 1766395735 1766395735 1766395735 MDT a|m|ctime 1766395735 1766395735 1766395735 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00226977 s, 451 kB/s oleg148-server: debugfs 1.47.3-wc2 (11-Nov-2025) STAT a|m|ctime 1766395735 1766395737 1766395737 MDT a|m|ctime 1766395735 1766395737 1766395737 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00616528 s, 166 kB/s oleg148-server: debugfs 1.47.3-wc2 (11-Nov-2025) STAT a|m|ctime 1766395735 1766395742 1766395742 MDT a|m|ctime 1766395735 1766395742 1766395742 oleg148-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg148-server: debugfs 1.47.3-wc2 (11-Nov-2025) STAT a|m|ctime 1766395735 1766395742 1766395742 MDT a|m|ctime 1766395735 1766395742 1766395742 oleg148-server: debugfs 1.47.3-wc2 (11-Nov-2025) STAT a|m|ctime 1766395735 1766395742 1766395751 MDT a|m|ctime 1766395735 1766395742 1766395751 PASS 104 (19s) == sanityn test 105: Glimpse and lock cancel race ======== 04:29:13 (1766395753) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d105.sanityn fail_loc=0x416 PASS 105 (18s) == sanityn test 106a: Verify the btime via statx() ======= 04:29:31 (1766395771) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00255823 s, 400 kB/s oleg148-server: debugfs 1.47.3-wc2 (11-Nov-2025) Inode: 5235 Type: regular Mode: 0644 Flags: 0x80000 Generation: 677684640 Version: 0x00000002:000186d9 User: 0 Group: 0 Project: 0 Size: 0 File ACL: 0 Links: 1 Blockcount: 0 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x69490f7b:00000000 -- Mon Dec 22 04:29:31 2025 atime: 0x69490f7b:00000000 -- Mon Dec 22 04:29:31 2025 mtime: 0x69490f7b:00000000 -- Mon Dec 22 04:29:31 2025 crtime: 0x69490f7c:4e709100 -- Mon Dec 22 04:29:32 2025 Size of extra inode fields: 32 Extended attributes: lma: fid=[0x200000bd1:0x1f41:0x0] compat=0 incompat=0 trusted.lov (56) = d0 0b d1 0b 01 00 00 00 41 1f 00 00 00 00 00 00 d1 0b 00 00 02 00 00 00 00 00 40 00 01 00 00 00 01 04 00 80 02 00 00 00 51 19 00 00 00 00 00 00 00 00 00 00 00 00 00 00 user.job (4) = "dd.0" linkea: idx=0 parent=[0x200000007:0x1:0x0] name='f106a.sanityn' trusted.som (24) = 04 00 00 00 00 00 00 00 00 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 EXTENTS: PASS 106a (2s) == sanityn test 106b: Glimpse RPCs test for statx ======== 04:29:33 (1766395773) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00460772 s, 222 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 8 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826942274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 04:29:34.000000000 -0500 Modify: 2025-12-22 04:29:34.000000000 -0500 Change: 2025-12-22 04:29:34.000000000 -0500 Birth: 2025-12-22 04:29:34.000000000 -0500 /mnt/lustre/f106b.sanityn 144115238826942274 -rw-r--r-- 1 0 0 1766395774 1766395774 1766395774 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 8 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826942274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 04:29:34.000000000 -0500 Modify: 2025-12-22 04:29:34.000000000 -0500 Change: 2025-12-22 04:29:34.000000000 -0500 Birth: 2025-12-22 04:29:34.000000000 -0500 1766395774 1024 8 PASS 106b (3s) == sanityn test 106c: Verify statx attributes mask ======= 04:29:36 (1766395776) PASS 106c (2s) == sanityn test 107a: Basic grouplock conflict =========== 04:29:38 (1766395778) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.122697 s, 85.5 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6436 0x1924 0x2c0000401 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.8196 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.8196 First grouplock blocks second one PASS 107a (5s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 04:29:43 (1766395783) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.118558 s, 88.4 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6483 0x1953 0x280000401 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.8196 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (8s) == sanityn test 108a: lseek: parallel updates ============ 04:29:51 (1766395791) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00206677 s, 4.8 MB/s llite.lustre-ffff9e1b82f6d000.hybrid_io=0 llite.lustre-ffff9e1b83169800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.19674 s, 2.0 MB/s PASS 108a (7s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 04:29:58 (1766395798) 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) Starting race between client mount instances (50 iterations): Iteration 1 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 2 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 3 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg148-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 4 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg148-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 5 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg148-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 6 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg148-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 7 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg148-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 8 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 9 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 10 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg148-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 11 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 12 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg148-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 13 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 14 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg148-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 15 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg148-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 16 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 17 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 18 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg148-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 19 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg148-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 20 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg148-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 21 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg148-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 22 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 23 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg148-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 24 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg148-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 25 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 26 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 27 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 28 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 29 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 30 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg148-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 31 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 32 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg148-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 33 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 34 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg148-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 35 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 36 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 37 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg148-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 38 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg148-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 39 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 40 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg148-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 41 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 42 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg148-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 43 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg148-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 44 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg148-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 45 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg148-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 46 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 47 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg148-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 48 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 49 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg148-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Iteration 50 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.148@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 PASS 109 (584s) == sanityn test 110: do not grant another lock on resend ========================================================== 04:39:42 (1766396382) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238843711491 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 04:39:42.000000000 -0500 Modify: 2025-12-22 04:39:42.000000000 -0500 Change: 2025-12-22 04:39:42.000000000 -0500 Birth: 2025-12-22 04:39:43.000000000 -0500 fail_loc=0 fail_val=0 Connected clients: oleg148-client.virtnet oleg148-client.virtnet PASS 110 (67s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 04:40:50 (1766396450) fail_loc=0x8000018a /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6784: 483575 Terminated $MULTIOP $DIR2/$tdir/mdt0dir/foodir/file2 Ow4096_c PASS 111 (9s) == sanityn test 112: update max-inherit in default LMV === 04:40:59 (1766396459) lmv.lustre-clilmv-ffff9e1b833a8800.qos_threshold_rr=90 lmv.lustre-clilmv-ffff9e1b8538b800.qos_threshold_rr=90 PASS 112 (7s) == sanityn test 113: check servers of specified fs ======= 04:41:06 (1766396466) lustre-OST0000-osc-ffff9e1b833a8800 active. lustre-OST0001-osc-ffff9e1b833a8800 active. lustre-OST0000-osc-ffff9e1b833a8800 active. lustre-OST0001-osc-ffff9e1b833a8800 active. MGC192.168.201.148@tcp active. PASS 113 (4s) == sanityn test 114: implicit default LMV inherit ======== 04:41:10 (1766396470) /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1 lmv_stripe_count: 2 lmv_stripe_offset: 2 lmv_hash_type: none lmv_max_inherit: 2 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1 lmv_stripe_count: 2 lmv_stripe_offset: 2 lmv_hash_type: none lmv_max_inherit: 2 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 PASS 114 (10s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 04:41:20 (1766396480) - create 2872 (time 1766396492.88 total 10.00 last 287.08) total: 3000 create in 10.45 seconds: 287.13 ops/second oleg148-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg148-server: htree_dump_int_block: Ext2 directory block not found while calculating checksum for htree root oleg148-server: USE: f345 f1829 fail_loc=0x80002401 fail_val=5 mkdir: cannot create directory '/mnt/lustre/d115.sanityn/f1829': File exists PASS 115 (22s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 04:41:42 (1766396502) llite.lustre-ffff9e1b833a8800.intent_mkdir=1 llite.lustre-ffff9e1b8538b800.intent_mkdir=1 MD layout /mnt/lustre/d116.sanityn: lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x200000bd0:0x802:0x0] 1 [0x240000404:0x798:0x0] mkdir /mnt/lustre/d116.sanityn/tdir0 setdirstripe -D -i1 /mnt/lustre2/d116.sanityn/tdir0 mkdir /mnt/lustre/d116.sanityn/tdir0/tdir11 lmv_stripe_count: 0 lmv_stripe_offset: 0 lmv_hash_type: none mdtidx FID[seq:oid:ver] lmv_stripe_count: 0 lmv_stripe_offset: 1 lmv_hash_type: none mdtidx FID[seq:oid:ver] PASS 116 (4s) == sanityn test 117a: TCU: Init and enable Trash Can on MDTs ========================================================== 04:41:46 (1766396506) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 PASS 117a (5s) == sanityn test 117b: Move regular file and empty dir into trash can dir ========================================================== 04:41:51 (1766396511) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain llite.lustre-ffff9e1b833a8800.trash_file_read=1 llite.lustre-ffff9e1b8538b800.trash_file_read=1 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.126385 s, 33.2 MB/s /mnt/lustre/.lustre/trash: MDT0000 MDT0001 /mnt/lustre/.lustre/trash/MDT0000: 0x200000bd2:0xbcc:0x0 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbcc:0x0: tdir1 tdir2 tf1 tf2 tf3 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbcc:0x0/tdir1: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbcc:0x0/tdir2: /mnt/lustre/.lustre/trash/MDT0001: llite.lustre-ffff9e1b833a8800.trash_file_read=0 llite.lustre-ffff9e1b8538b800.trash_file_read=0 mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117b (8s) == sanityn test 117c: Move deleted tree with multiple levels into trash ========================================================== 04:41:59 (1766396519) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain /mnt/lustre/.lustre/trash/MDT0000: 0x200000bd2:0xbcc:0x0 0x200000bd2:0xbd2:0x0 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbcc:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbd2:0x0: tdir1 tdir2 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbd2:0x0/tdir1: tdir11 tdir12 tfile13 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbd2:0x0/tdir1/tdir11: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbd2:0x0/tdir1/tdir12: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbd2:0x0/tdir2: mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117c (8s) == sanityn test 117d: Per-User Trash can Type testing ==== 04:42:07 (1766396527) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid Trash Can Type: uid uid running as uid/gid/euid/egid 500/500/500/500, groups: 500 [mkdir] [/mnt/lustre/d117d.sanityn/tdir2] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [mkdir] [/mnt/lustre/d117d.sanityn/tdir2/tdir21] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [mkdir] [/mnt/lustre/d117d.sanityn/tdir2/tdir22] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d117d.sanityn/tdir2/tfile23] uid1: 0 uid2: 500 /mnt/lustre/.lustre/trash/MDT0000: 0 0x200000bd2:0xbcc:0x0 0x200000bd2:0xbd2:0x0 500 /mnt/lustre/.lustre/trash/MDT0000/0: 0x200000bd2:0xbdb:0x0 /mnt/lustre/.lustre/trash/MDT0000/0/0x200000bd2:0xbdb:0x0: tdir11 tdir12 tfile13 /mnt/lustre/.lustre/trash/MDT0000/0/0x200000bd2:0xbdb:0x0/tdir11: /mnt/lustre/.lustre/trash/MDT0000/0/0x200000bd2:0xbdb:0x0/tdir12: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbcc:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbd2:0x0: /mnt/lustre/.lustre/trash/MDT0000/500: 0x200000bd2:0xbdf:0x0 /mnt/lustre/.lustre/trash/MDT0000/500/0x200000bd2:0xbdf:0x0: tdir21 tdir22 tfile23 /mnt/lustre/.lustre/trash/MDT0000/500/0x200000bd2:0xbdf:0x0/tdir21: /mnt/lustre/.lustre/trash/MDT0000/500/0x200000bd2:0xbdf:0x0/tdir22: mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117d (8s) == sanityn test 117e: Undeleted dir in trash should keep its original xattrs ========================================================== 04:42:15 (1766396535) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain /mnt/lustre/.lustre/trash/MDT0000: 0 0x200000bd2:0xbcc:0x0 0x200000bd2:0xbd2:0x0 0x200000bd2:0xbe3:0x0 500 /mnt/lustre/.lustre/trash/MDT0000/0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbcc:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbd2:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe3:0x0: tdir1 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe3:0x0/tdir1: tdir11 tfile12 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe3:0x0/tdir1/tdir11: /mnt/lustre/.lustre/trash/MDT0000/500: trusted.name11="value11" trusted.name12="value12" trusted.name1="value1" mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117e (7s) == sanityn test 117f: Uncache the dentry under the trash dir ========================================================== 04:42:22 (1766396542) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain /mnt/lustre/.lustre/trash/MDT0000: 0 0x200000bd2:0xbcc:0x0 0x200000bd2:0xbd2:0x0 0x200000bd2:0xbe3:0x0 0x200000bd2:0xbe7:0x0 500 /mnt/lustre/.lustre/trash/MDT0000/0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbcc:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbd2:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe7:0x0: tdir1 tfile3 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe7:0x0/tdir1: /mnt/lustre/.lustre/trash/MDT0000/500: tdir1 tfile3 /mnt/lustre/.lustre/trash/MDT0000: 0 0x200000bd2:0xbcc:0x0 0x200000bd2:0xbd2:0x0 0x200000bd2:0xbe3:0x0 0x200000bd2:0xbe7:0x0 500 /mnt/lustre/.lustre/trash/MDT0000/0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbcc:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbd2:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe7:0x0: tdir1 tdir2 tfile3 tfile4 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe7:0x0/tdir1: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe7:0x0/tdir2: /mnt/lustre/.lustre/trash/MDT0000/500: tdir2 tfile4 -u-------------------- /mnt/lustre/.lustre/trash/MDT0000/0 /mnt/lustre/.lustre/trash/MDT0000/0: -u-------------------- /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe7:0x0 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe7:0x0: -u-------------------- /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe7:0x0/tdir2 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe7:0x0/tdir2: -u------------e------- /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe7:0x0/tfile4 -u-------------------- /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe7:0x0/tdir1 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe7:0x0/tdir1: -u------------e------- /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe7:0x0/tfile3 -u-------------------- /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbcc:0x0 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbcc:0x0: -u-------------------- /mnt/lustre/.lustre/trash/MDT0000/500 /mnt/lustre/.lustre/trash/MDT0000/500: -u-------------------- /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe3:0x0 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe3:0x0: -u-------------------- /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbd2:0x0 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbd2:0x0: mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117f (8s) == sanityn test 117g: Access .Trash for a non-striped directory ========================================================== 04:42:30 (1766396550) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain /mnt/lustre/.lustre/trash/MDT0000: 0 0x200000bd2:0xbcc:0x0 0x200000bd2:0xbd2:0x0 0x200000bd2:0xbe3:0x0 0x200000bd2:0xbe7:0x0 0x200000bd2:0xbec:0x0 500 /mnt/lustre/.lustre/trash/MDT0000/0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbcc:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbd2:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe7:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbec:0x0: tdir1 tfile3 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbec:0x0/tdir1: /mnt/lustre/.lustre/trash/MDT0000/500: ==> check '.Trash': tdir1 tfile3 <== tdir1 tfile3 tdir1 tfile3 tdir21 tfile22 tdir2 tdir21 tfile22 mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117g (7s) == sanityn test 117h: Permission testing of various operations for files in Trash Can ========================================================== 04:42:37 (1766396557) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain Trash list: ===> /mnt/lustre/.lustre/trash/MDT0000: 0 0x200000bd2:0xbcc:0x0 0x200000bd2:0xbd2:0x0 0x200000bd2:0xbe3:0x0 0x200000bd2:0xbe7:0x0 0x200000bd2:0xbec:0x0 0x200000bd2:0xbf3:0x0 500 /mnt/lustre/.lustre/trash/MDT0000/0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbcc:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbd2:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe7:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbec:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbf3:0x0: tdir1 tfile3 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbf3:0x0/tdir1: /mnt/lustre/.lustre/trash/MDT0000/500: <=== touch: setting times of '/mnt/lustre/.lustre/trash/f117h.sanityn.crfail': No such file or directory mkdir: cannot create directory '/mnt/lustre/.lustre/trash/f117h.sanityn.crfail': Operation not permitted touch: setting times of '/mnt/lustre/.lustre/trash/MDT0000/f117h.sanityn.crfail': No such file or directory mkdir: cannot create directory '/mnt/lustre/.lustre/trash/MDT0000/f117h.sanityn.crfail': Operation not permitted rmdir: failed to remove '/mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbf3:0x0': Operation not permitted touch: setting times of '/mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbf3:0x0/f117h.sanityn.crfail': No such file or directory mv: cannot move '/mnt/lustre/d117h.sanityn/f117h.sanityn.mvfail' to '/mnt/lustre/.lustre/trash/f117h.sanityn.mvfail': Operation not permitted mv: cannot move '/mnt/lustre/d117h.sanityn/f117h.sanityn.mvfail' to '/mnt/lustre/.lustre/trash/MDT0000/f117h.sanityn.mvfail': Operation not permitted mv: cannot move '/mnt/lustre/d117h.sanityn/f117h.sanityn.mvfail' to '/mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbf3:0x0/f117h.sanityn.mvfail': Operation not permitted mv: cannot move '/mnt/lustre/d117h.sanityn/d117h.sanityn.mvfail' to '/mnt/lustre/.lustre/trash/d117h.sanityn.mvfail': Operation not permitted mv: cannot move '/mnt/lustre/d117h.sanityn/d117h.sanityn.mvfail' to '/mnt/lustre/.lustre/trash/MDT0000/d117h.sanityn.mvfail': Operation not permitted mv: cannot move '/mnt/lustre/d117h.sanityn/d117h.sanityn.mvfail' to '/mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbf3:0x0/d117h.sanityn.mvfail': Operation not permitted chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbf3:0x0//tfile3 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbf3:0x0//tfile3 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbf3:0x0//tfile3 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbf3:0x0//tfile3 chmod: changing permissions of '/mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbf3:0x0//tfile3': Bad file descriptor truncate: cannot truncate '/mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbf3:0x0//tfile3' to length 4096: Bad file descriptor chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbf3:0x0//tdir1 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbf3:0x0//tdir1 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbf3:0x0//tdir1 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbf3:0x0//tdir1 chmod: changing permissions of '/mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbf3:0x0//tdir1': Bad file descriptor setfattr: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbf3:0x0//tfile3: Bad file descriptor setfattr: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbf3:0x0//tfile3: Bad file descriptor setfattr: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbf3:0x0//tdir1: Bad file descriptor setfattr: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbf3:0x0//tdir1: Bad file descriptor open: Bad file descriptor open: Bad file descriptor read: Operation not permitted mmap: Operation not permitted open: Is a directory open: Is a directory mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117h (7s) == sanityn test 117i: Forbid to create a file with name .Trash ========================================================== 04:42:44 (1766396564) mkdir: cannot create directory '/mnt/lustre/d117i.sanityn/.Trash': Invalid argument touch: setting times of '/mnt/lustre/d117i.sanityn/.Trash': No such file or directory open(O_RDWR|O_CREAT): Invalid argument mcreate: cannot create `/mnt/lustre/d117i.sanityn/.Trash' with mode 0100644: Invalid argument ln: failed to create hard link '/mnt/lustre/d117i.sanityn/.Trash' => '/mnt/lustre/d117i.sanityn/f117i.sanityn': Invalid argument ln: failed to create symbolic link '/mnt/lustre/d117i.sanityn/.Trash': Invalid argument mv: cannot move '/mnt/lustre/d117i.sanityn/f117i.sanityn' to a subdirectory of itself, '/mnt/lustre/d117i.sanityn/.Trash' mv: cannot move '/mnt/lustre/d117i.sanityn/f117i.sanityn' to a subdirectory of itself, '/mnt/lustre/.Trash' ln: failed to create symbolic link '/mnt/lustre/d117i.sanityn/.Trash': Invalid argument mv: cannot move '/mnt/lustre/d117i.sanityn/d117i.sanityn' to a subdirectory of itself, '/mnt/lustre/d117i.sanityn/.Trash' mv: cannot move '/mnt/lustre/d117i.sanityn/d117i.sanityn' to a subdirectory of itself, '/mnt/lustre/.Trash' PASS 117i (4s) == sanityn test 117j: Move back file from Trash Can ====== 04:42:48 (1766396568) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain tfile.1 -u------------e------- /mnt/lustre/d117j.sanityn/.Trash/tfile.1 --------------e------- /mnt/lustre/d117j.sanityn/tfile.1 List /mnt/lustre/d117j.sanityn/.Trash /mnt/lustre/d117j.sanityn/.Trash: tdir1 /mnt/lustre/d117j.sanityn/.Trash/tdir1: tdir11 tfile12 tfile13 /mnt/lustre/d117j.sanityn/.Trash/tdir1/tdir11: -u-------------------- /mnt/lustre/d117j.sanityn/.Trash/tdir1 /mnt/lustre/d117j.sanityn/.Trash/tdir1: -u------------e------- /mnt/lustre/d117j.sanityn/.Trash/tdir1/tfile13 -u------------e------- /mnt/lustre/d117j.sanityn/.Trash/tdir1/tfile12 -u-------------------- /mnt/lustre/d117j.sanityn/.Trash/tdir1/tdir11 /mnt/lustre/d117j.sanityn/.Trash/tdir1/tdir11: === DONE === mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117j (7s) == sanityn test 117k: Unrm file from Trash Can =========== 04:42:55 (1766396575) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain ==== DONE ==== mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117k (8s) == sanityn test 117l: Clean up files in Trash Can via command 'lfs trash clean' ========================================================== 04:43:03 (1766396583) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain Phase 0/PR: remove dir 1:0x200000bd2:0xc0c:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd2:0xc0c:0x0 from Trash Can Skip stub dir 1:0x200000bd2:0xc0c:0x0 Phase 1/EN: remove dir 1:0x200000bd2:0xc0c:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd2:0xbec:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd2:0xbec:0x0 from Trash Can Skip stub dir 1:0x200000bd2:0xbec:0x0 Phase 1/EN: remove dir 1:0x200000bd2:0xbec:0x0 from Trash Can Phase 0/PR: remove dir 1:0 from Trash Can Phase 2/PO: remove dir 1:0 from Trash Can Phase 1/EN: remove dir 1:0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd2:0xbe7:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd2:0xbe7:0x0 from Trash Can Skip stub dir 1:0x200000bd2:0xbe7:0x0 Phase 1/EN: remove dir 1:0x200000bd2:0xbe7:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd2:0xbf3:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd2:0xbf3:0x0 from Trash Can Skip stub dir 1:0x200000bd2:0xbf3:0x0 Phase 1/EN: remove dir 1:0x200000bd2:0xbf3:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd2:0xbcc:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd2:0xbcc:0x0 from Trash Can Skip stub dir 1:0x200000bd2:0xbcc:0x0 Phase 1/EN: remove dir 1:0x200000bd2:0xbcc:0x0 from Trash Can Phase 0/PR: remove dir 1:500 from Trash Can Phase 2/PO: remove dir 1:500 from Trash Can Phase 1/EN: remove dir 1:500 from Trash Can Phase 0/PR: remove dir 1:0x200000bd2:0xc12:0x0 from Trash Can Phase 0/PR: remove dir 2:tdir1 from Trash Can Phase 1/EN: remove file 3:tfile13 from Trash Can Phase 1/EN: remove file 3:tfile12 from Trash Can Phase 0/PR: remove dir 3:tdir11 from Trash Can Phase 2/PO: remove dir 3:tdir11 from Trash Can Phase 1/EN: remove dir 3:tdir11 from Trash Can Phase 2/PO: remove dir 2:tdir1 from Trash Can Phase 1/EN: remove dir 2:tdir1 from Trash Can Phase 2/PO: remove dir 1:0x200000bd2:0xc12:0x0 from Trash Can Skip stub dir 1:0x200000bd2:0xc12:0x0 Phase 1/EN: remove dir 1:0x200000bd2:0xc12:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd2:0xbe3:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd2:0xbe3:0x0 from Trash Can Skip stub dir 1:0x200000bd2:0xbe3:0x0 Phase 1/EN: remove dir 1:0x200000bd2:0xbe3:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd2:0xbd2:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd2:0xbd2:0x0 from Trash Can Skip stub dir 1:0x200000bd2:0xbd2:0x0 Phase 1/EN: remove dir 1:0x200000bd2:0xbd2:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd2:0xc06:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd2:0xc06:0x0 from Trash Can Skip stub dir 1:0x200000bd2:0xc06:0x0 Phase 1/EN: remove dir 1:0x200000bd2:0xc06:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd2:0xc0c:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd2:0xc0c:0x0 from Trash Can Skip stub dir 1:0x200000bd2:0xc0c:0x0 Phase 1/EN: remove dir 1:0x200000bd2:0xc0c:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd2:0xbec:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd2:0xbec:0x0 from Trash Can Skip stub dir 1:0x200000bd2:0xbec:0x0 Phase 1/EN: remove dir 1:0x200000bd2:0xbec:0x0 from Trash Can Phase 0/PR: remove dir 1:0 from Trash Can Phase 2/PO: remove dir 1:0 from Trash Can Phase 1/EN: remove dir 1:0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd2:0xbe7:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd2:0xbe7:0x0 from Trash Can Skip stub dir 1:0x200000bd2:0xbe7:0x0 Phase 1/EN: remove dir 1:0x200000bd2:0xbe7:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd2:0xbf3:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd2:0xbf3:0x0 from Trash Can Skip stub dir 1:0x200000bd2:0xbf3:0x0 Phase 1/EN: remove dir 1:0x200000bd2:0xbf3:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd2:0xbcc:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd2:0xbcc:0x0 from Trash Can Skip stub dir 1:0x200000bd2:0xbcc:0x0 Phase 1/EN: remove dir 1:0x200000bd2:0xbcc:0x0 from Trash Can Phase 0/PR: remove dir 1:500 from Trash Can Phase 2/PO: remove dir 1:500 from Trash Can Phase 1/EN: remove dir 1:500 from Trash Can Phase 0/PR: remove dir 1:0x200000bd2:0xc12:0x0 from Trash Can Phase 0/PR: remove dir 2:tdir1 from Trash Can Phase 1/EN: remove file 3:tfile13 from Trash Can Phase 1/EN: remove file 3:tfile12 from Trash Can Phase 0/PR: remove dir 3:tdir11 from Trash Can Phase 2/PO: remove dir 3:tdir11 from Trash Can Phase 1/EN: remove dir 3:tdir11 from Trash Can Phase 2/PO: remove dir 2:tdir1 from Trash Can Phase 1/EN: remove dir 2:tdir1 from Trash Can Phase 2/PO: remove dir 1:0x200000bd2:0xc12:0x0 from Trash Can Skip stub dir 1:0x200000bd2:0xc12:0x0 Phase 1/EN: remove dir 1:0x200000bd2:0xc12:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd2:0xc1c:0x0 from Trash Can Phase 1/EN: remove file 2:tfile22 from Trash Can Phase 2/PO: remove dir 1:0x200000bd2:0xc1c:0x0 from Trash Can Skip stub dir 1:0x200000bd2:0xc1c:0x0 Phase 1/EN: remove dir 1:0x200000bd2:0xc1c:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd2:0xbe3:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd2:0xbe3:0x0 from Trash Can Skip stub dir 1:0x200000bd2:0xbe3:0x0 Phase 1/EN: remove dir 1:0x200000bd2:0xbe3:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd2:0xbd2:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd2:0xbd2:0x0 from Trash Can Skip stub dir 1:0x200000bd2:0xbd2:0x0 Phase 1/EN: remove dir 1:0x200000bd2:0xbd2:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd2:0xc06:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd2:0xc06:0x0 from Trash Can Skip stub dir 1:0x200000bd2:0xc06:0x0 Phase 1/EN: remove dir 1:0x200000bd2:0xc06:0x0 from Trash Can mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117l (9s) == sanityn test 117m: Empty Trash Can via the command 'lfs trash clean' ========================================================== 04:43:12 (1766396592) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain /mnt/lustre/.lustre/trash: MDT0000 MDT0001 /mnt/lustre/.lustre/trash/MDT0000: 0 0x200000bd2:0xbcc:0x0 0x200000bd2:0xbd2:0x0 0x200000bd2:0xbe3:0x0 0x200000bd2:0xbe7:0x0 0x200000bd2:0xbec:0x0 0x200000bd2:0xbf3:0x0 0x200000bd2:0xc06:0x0 0x200000bd2:0xc0c:0x0 0x200000bd2:0xc12:0x0 0x200000bd2:0xc1c:0x0 0x200000bd2:0xc1f:0x0 0x200000bd2:0xc25:0x0 500 /mnt/lustre/.lustre/trash/MDT0000/0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbcc:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbd2:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe7:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbec:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbf3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc06:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc0c:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc12:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc1c:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc1f:0x0: tdir1 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc1f:0x0/tdir1: tdir11 tfile12 tfile13 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc1f:0x0/tdir1/tdir11: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc25:0x0: tfile22 /mnt/lustre/.lustre/trash/MDT0000/500: /mnt/lustre/.lustre/trash/MDT0001: Phase 0/PR: remove dir 1:0x200000bd2:0xc0c:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd2:0xc0c:0x0 from Trash Can Skip stub dir 1:0x200000bd2:0xc0c:0x0 Phase 1/EN: remove dir 1:0x200000bd2:0xc0c:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd2:0xbec:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd2:0xbec:0x0 from Trash Can Skip stub dir 1:0x200000bd2:0xbec:0x0 Phase 1/EN: remove dir 1:0x200000bd2:0xbec:0x0 from Trash Can Phase 0/PR: remove dir 1:0 from Trash Can Phase 2/PO: remove dir 1:0 from Trash Can Phase 1/EN: remove dir 1:0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd2:0xbe7:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd2:0xbe7:0x0 from Trash Can Skip stub dir 1:0x200000bd2:0xbe7:0x0 Phase 1/EN: remove dir 1:0x200000bd2:0xbe7:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd2:0xbf3:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd2:0xbf3:0x0 from Trash Can Skip stub dir 1:0x200000bd2:0xbf3:0x0 Phase 1/EN: remove dir 1:0x200000bd2:0xbf3:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd2:0xbcc:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd2:0xbcc:0x0 from Trash Can Skip stub dir 1:0x200000bd2:0xbcc:0x0 Phase 1/EN: remove dir 1:0x200000bd2:0xbcc:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd2:0xc1f:0x0 from Trash Can Phase 0/PR: remove dir 2:tdir1 from Trash Can Phase 1/EN: remove file 3:tfile13 from Trash Can Phase 1/EN: remove file 3:tfile12 from Trash Can Phase 0/PR: remove dir 3:tdir11 from Trash Can Phase 2/PO: remove dir 3:tdir11 from Trash Can Phase 1/EN: remove dir 3:tdir11 from Trash Can Phase 2/PO: remove dir 2:tdir1 from Trash Can Phase 1/EN: remove dir 2:tdir1 from Trash Can Phase 2/PO: remove dir 1:0x200000bd2:0xc1f:0x0 from Trash Can Skip stub dir 1:0x200000bd2:0xc1f:0x0 Phase 1/EN: remove dir 1:0x200000bd2:0xc1f:0x0 from Trash Can Phase 0/PR: remove dir 1:500 from Trash Can Phase 2/PO: remove dir 1:500 from Trash Can Phase 1/EN: remove dir 1:500 from Trash Can Phase 0/PR: remove dir 1:0x200000bd2:0xc25:0x0 from Trash Can Phase 1/EN: remove file 2:tfile22 from Trash Can Phase 2/PO: remove dir 1:0x200000bd2:0xc25:0x0 from Trash Can Skip stub dir 1:0x200000bd2:0xc25:0x0 Phase 1/EN: remove dir 1:0x200000bd2:0xc25:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd2:0xc12:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd2:0xc12:0x0 from Trash Can Skip stub dir 1:0x200000bd2:0xc12:0x0 Phase 1/EN: remove dir 1:0x200000bd2:0xc12:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd2:0xc1c:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd2:0xc1c:0x0 from Trash Can Skip stub dir 1:0x200000bd2:0xc1c:0x0 Phase 1/EN: remove dir 1:0x200000bd2:0xc1c:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd2:0xbe3:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd2:0xbe3:0x0 from Trash Can Skip stub dir 1:0x200000bd2:0xbe3:0x0 Phase 1/EN: remove dir 1:0x200000bd2:0xbe3:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd2:0xbd2:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd2:0xbd2:0x0 from Trash Can Skip stub dir 1:0x200000bd2:0xbd2:0x0 Phase 1/EN: remove dir 1:0x200000bd2:0xbd2:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd2:0xc06:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd2:0xc06:0x0 from Trash Can Skip stub dir 1:0x200000bd2:0xc06:0x0 Phase 1/EN: remove dir 1:0x200000bd2:0xc06:0x0 from Trash Can mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117m (10s) == sanityn test 117n: Update LinkEA accordingly when moving file into Trash Can ========================================================== 04:43:22 (1766396602) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain /mnt/lustre/d117n.sanityn/.Trash//tfile.1: count 1 0: pfid [0x200000bd0:0x81c:0x0], name 'tfile.1' /mnt/lustre/d117n.sanityn/.Trash//tfile.1: count 1 0: pfid [0x200000bd0:0x81c:0x0], name 'tfile.1' /mnt/lustre/d117n.sanityn/.Trash//tfile.1: count 1 0: pfid [0x200000bd0:0x81c:0x0], name 'tfile.1' /mnt/lustre/d117n.sanityn/.Trash//tfile.1: count 1 0: pfid [0x200000bd0:0x81c:0x0], name 'tfile.1' /mnt/lustre/d117n.sanityn/.Trash//tdir1/tdir11: count 1 0: pfid [0x200000bd0:0x81d:0x0], name 'tdir11' /mnt/lustre/d117n.sanityn/.Trash//tdir1/tdir11: count 1 0: pfid [0x200000bd0:0x81d:0x0], name 'tdir11' /mnt/lustre/d117n.sanityn/.Trash//tdir1/tdir11: count 1 0: pfid [0x200000bd0:0x81d:0x0], name 'tdir11' /mnt/lustre/d117n.sanityn/.Trash//tdir1/tdir11: count 1 0: pfid [0x200000bd0:0x81d:0x0], name 'tdir11' /mnt/lustre/d117n.sanityn/.Trash//tdir1/tfile12: count 1 0: pfid [0x200000bd0:0x81d:0x0], name 'tfile12' /mnt/lustre/d117n.sanityn/.Trash//tdir1/tfile12: count 1 0: pfid [0x200000bd0:0x81d:0x0], name 'tfile12' /mnt/lustre/d117n.sanityn/.Trash//tdir1/tfile12: count 1 0: pfid [0x200000bd0:0x81d:0x0], name 'tfile12' /mnt/lustre/d117n.sanityn/.Trash//tdir1/tfile12: count 1 0: pfid [0x200000bd0:0x81d:0x0], name 'tfile12' /mnt/lustre/d117n.sanityn/.Trash//tdir1/tfile13: count 1 0: pfid [0x200000bd0:0x81d:0x0], name 'tfile13' /mnt/lustre/d117n.sanityn/.Trash//tdir1/tfile13: count 1 0: pfid [0x200000bd0:0x81d:0x0], name 'tfile13' /mnt/lustre/d117n.sanityn/.Trash//tdir1/tfile13: count 1 0: pfid [0x200000bd0:0x81d:0x0], name 'tfile13' /mnt/lustre/d117n.sanityn/.Trash//tdir1/tfile13: count 1 0: pfid [0x200000bd0:0x81d:0x0], name 'tfile13' /mnt/lustre/d117n.sanityn/.Trash//tdir1: count 1 0: pfid [0x200000bd0:0x81c:0x0], name 'tdir1' /mnt/lustre/d117n.sanityn/.Trash//tdir1: count 1 0: pfid [0x200000bd0:0x81c:0x0], name 'tdir1' /mnt/lustre/d117n.sanityn/.Trash//tdir1: count 1 0: pfid [0x200000bd0:0x81c:0x0], name 'tdir1' /mnt/lustre/d117n.sanityn/.Trash//tdir1: count 1 0: pfid [0x200000bd0:0x81c:0x0], name 'tdir1' /mnt/lustre/d117n.sanityn/tdir2/.Trash//tfile22: count 1 0: pfid [0x200000bd0:0x81e:0x0], name 'tfile22' /mnt/lustre/d117n.sanityn/tdir2/.Trash//tfile22: count 1 0: pfid [0x200000bd0:0x81e:0x0], name 'tfile22' /mnt/lustre/d117n.sanityn/tdir2/.Trash//tfile22: count 1 0: pfid [0x200000bd0:0x81e:0x0], name 'tfile22' /mnt/lustre/d117n.sanityn/tdir2/.Trash//tfile22: count 1 0: pfid [0x200000bd0:0x81e:0x0], name 'tfile22' mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117n (10s) == sanityn test 117o: Set/Clear NOTRASH flag on file ===== 04:43:32 (1766396612) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain Set flags 0X80000/0X80040 OK: rc=0 file: /mnt/lustre/d117o.sanityn/tfile.1, flags: 0X80040 notrash, uid: 0 gid: 0 projid: 0 timestamp: 1766396616 file: /mnt/lustre/d117o.sanityn/tfile.1, flags: 0X80040 notrash, uid: 0 gid: 0 projid: 0 timestamp: 1766396616 file: /mnt/lustre/d117o.sanityn/tdir1, flags: 0X40 notrash, uid: 0 gid: 0 projid: 0 timestamp: 1766396616 file: /mnt/lustre/d117o.sanityn/tdir1/tdir11, flags: 0X40 notrash, uid: 0 gid: 0 projid: 0 timestamp: 1766396617 file: /mnt/lustre/d117o.sanityn/tdir1, flags: 0X40 notrash, uid: 0 gid: 0 projid: 0 timestamp: 1766396617 file: /mnt/lustre/d117o.sanityn/tdir1/tdir11, flags: 0X40 notrash, uid: 0 gid: 0 projid: 0 timestamp: 1766396617 file: /mnt/lustre/d117o.sanityn/tdir1/tfile12, flags: 0X80040 notrash, uid: 0 gid: 0 projid: 0 timestamp: 1766396617 lsattr /mnt/lustre/d117o.sanityn/tdir1: ------d-------e------- /mnt/lustre/d117o.sanityn/tdir1/tfile12 ------d--------------- /mnt/lustre/d117o.sanityn/tdir1/tdir11 lsattr /mnt/lustre/d117o.sanityn/tdir1/tdir11: lsattr /mnt/lustre/d117o.sanityn/tdir1/tfile12: ------d-------e------- /mnt/lustre/d117o.sanityn/tdir1/tfile12 file: /mnt/lustre/d117o.sanityn/tdir1/tdir11, flags: 0X40 notrash, uid: 0 gid: 0 projid: 0 timestamp: 1766396617 file: /mnt/lustre/d117o.sanityn/tdir1/tfile12, flags: 0X80040 notrash, uid: 0 gid: 0 projid: 0 timestamp: 1766396617 file: /mnt/lustre/d117o.sanityn/tdir1/tfile13, flags: 0X80040 notrash, uid: 0 gid: 0 projid: 0 timestamp: 1766396617 Set flags 0X80040/0X80000 OK: rc=0 chattr 1.47.3-wc2 (11-Nov-2025) Flags of /mnt/lustre/d117o.sanityn/tdir1 set as ---------------------- file: /mnt/lustre/d117o.sanityn/tdir1/tfile13, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396617 tfile13 mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117o (10s) == sanityn test 117p: Trash Can handling for remote dir/file ========================================================== 04:43:42 (1766396622) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain /mnt/lustre/d117p.sanityn/d117p.sanityn.1/.Trash/f117p.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6460 0x193c 0x2c0000401 /mnt/lustre/.lustre/trash: MDT0000 MDT0001 /mnt/lustre/.lustre/trash/MDT0000: 0 0x200000bd2:0xbcc:0x0 0x200000bd2:0xbd2:0x0 0x200000bd2:0xbe3:0x0 0x200000bd2:0xbe7:0x0 0x200000bd2:0xbec:0x0 0x200000bd2:0xbf3:0x0 0x200000bd2:0xc06:0x0 0x200000bd2:0xc0c:0x0 0x200000bd2:0xc12:0x0 0x200000bd2:0xc1c:0x0 0x200000bd2:0xc1f:0x0 0x200000bd2:0xc25:0x0 0x200000bd2:0xc28:0x0 0x200000bd2:0xc2e:0x0 0x200000bd2:0xc33:0x0 500 /mnt/lustre/.lustre/trash/MDT0000/0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbcc:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbd2:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe7:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbec:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbf3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc06:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc0c:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc12:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc1c:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc1f:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc25:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc28:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc2e:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc33:0x0: /mnt/lustre/.lustre/trash/MDT0000/500: /mnt/lustre/.lustre/trash/MDT0001: 0x240000405:0xc:0x0 /mnt/lustre/.lustre/trash/MDT0001/0x240000405:0xc:0x0: f117p.sanityn /mnt/lustre/.lustre/trash: MDT0000 MDT0001 /mnt/lustre/.lustre/trash/MDT0000: 0 0x200000bd2:0xbcc:0x0 0x200000bd2:0xbd2:0x0 0x200000bd2:0xbe3:0x0 0x200000bd2:0xbe7:0x0 0x200000bd2:0xbec:0x0 0x200000bd2:0xbf3:0x0 0x200000bd2:0xc06:0x0 0x200000bd2:0xc0c:0x0 0x200000bd2:0xc12:0x0 0x200000bd2:0xc1c:0x0 0x200000bd2:0xc1f:0x0 0x200000bd2:0xc25:0x0 0x200000bd2:0xc28:0x0 0x200000bd2:0xc2e:0x0 0x200000bd2:0xc33:0x0 0x200000bd2:0xc37:0x0 500 /mnt/lustre/.lustre/trash/MDT0000/0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbcc:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbd2:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe7:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbec:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbf3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc06:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc0c:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc12:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc1c:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc1f:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc25:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc28:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc2e:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc33:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc37:0x0: d117p.sanityn.01 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc37:0x0/d117p.sanityn.01: /mnt/lustre/.lustre/trash/MDT0000/500: /mnt/lustre/.lustre/trash/MDT0001: 0x240000405:0xc:0x0 /mnt/lustre/.lustre/trash/MDT0001/0x240000405:0xc:0x0: f117p.sanityn mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117p (11s) == sanityn test 117pb: Trash Can handling for remote dir/file with multiple levels ========================================================== 04:43:53 (1766396633) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain /mnt/lustre/.lustre/trash: MDT0000 MDT0001 /mnt/lustre/.lustre/trash/MDT0000: 0 0x200000bd2:0xbcc:0x0 0x200000bd2:0xbd2:0x0 0x200000bd2:0xbe3:0x0 0x200000bd2:0xbe7:0x0 0x200000bd2:0xbec:0x0 0x200000bd2:0xbf3:0x0 0x200000bd2:0xc06:0x0 0x200000bd2:0xc0c:0x0 0x200000bd2:0xc12:0x0 0x200000bd2:0xc1c:0x0 0x200000bd2:0xc1f:0x0 0x200000bd2:0xc25:0x0 0x200000bd2:0xc28:0x0 0x200000bd2:0xc2e:0x0 0x200000bd2:0xc33:0x0 0x200000bd2:0xc37:0x0 0x200000bd2:0xc39:0x0 500 /mnt/lustre/.lustre/trash/MDT0000/0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbcc:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbd2:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe7:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbec:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbf3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc06:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc0c:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc12:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc1c:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc1f:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc25:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc28:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc2e:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc33:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc37:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc39:0x0: d117pb.sanityn.0 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc39:0x0/d117pb.sanityn.0: d117pb.sanityn.01 f117pb.sanityn.02 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc39:0x0/d117pb.sanityn.0/d117pb.sanityn.01: /mnt/lustre/.lustre/trash/MDT0000/500: /mnt/lustre/.lustre/trash/MDT0001: 0x240000405:0xc:0x0 /mnt/lustre/.lustre/trash/MDT0001/0x240000405:0xc:0x0: mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117pb (8s) == sanityn test 117q: Access Trash Can via '.Trash' for striped directory ========================================================== 04:44:01 (1766396641) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain #0 -> 0x200000bd0:0x821:0x0 #1 -> 0x240000404:0x799:0x0 file.1 file.2 Creating 100 files under striped directory: /mnt/lustre/d117q.sanityn/striped_dir total: 100 open/close in 1.11 seconds: 90.39 ops/second f117q.sanityn0 -> 0 f117q.sanityn1 -> 0 f117q.sanityn2 -> 0 f117q.sanityn3 -> 0 f117q.sanityn4 -> 0 f117q.sanityn5 -> 0 f117q.sanityn6 -> 0 f117q.sanityn7 -> 0 f117q.sanityn8 -> 0 f117q.sanityn9 -> 0 f117q.sanityn10 -> 0 f117q.sanityn11 -> 0 f117q.sanityn12 -> 0 f117q.sanityn13 -> 0 f117q.sanityn14 -> 1 f117q.sanityn15 -> 1 f117q.sanityn16 -> 0 f117q.sanityn17 -> 1 f117q.sanityn18 -> 0 f117q.sanityn19 -> 1 f117q.sanityn20 -> 1 f117q.sanityn21 -> 1 f117q.sanityn22 -> 1 f117q.sanityn23 -> 1 f117q.sanityn24 -> 1 f117q.sanityn25 -> 0 f117q.sanityn26 -> 1 f117q.sanityn27 -> 0 f117q.sanityn28 -> 1 f117q.sanityn29 -> 1 f117q.sanityn30 -> 1 f117q.sanityn31 -> 1 f117q.sanityn32 -> 0 f117q.sanityn33 -> 1 f117q.sanityn34 -> 0 f117q.sanityn35 -> 0 f117q.sanityn36 -> 1 f117q.sanityn37 -> 0 f117q.sanityn38 -> 1 f117q.sanityn39 -> 0 f117q.sanityn40 -> 1 f117q.sanityn41 -> 0 f117q.sanityn42 -> 1 f117q.sanityn43 -> 1 f117q.sanityn44 -> 0 f117q.sanityn45 -> 0 f117q.sanityn46 -> 0 f117q.sanityn47 -> 1 f117q.sanityn48 -> 1 f117q.sanityn49 -> 0 f117q.sanityn50 -> 0 f117q.sanityn51 -> 0 f117q.sanityn52 -> 1 f117q.sanityn53 -> 1 f117q.sanityn54 -> 1 f117q.sanityn55 -> 0 f117q.sanityn56 -> 1 f117q.sanityn57 -> 1 f117q.sanityn58 -> 1 f117q.sanityn59 -> 0 f117q.sanityn60 -> 1 f117q.sanityn61 -> 1 f117q.sanityn62 -> 1 f117q.sanityn63 -> 1 f117q.sanityn64 -> 1 f117q.sanityn65 -> 0 f117q.sanityn66 -> 1 f117q.sanityn67 -> 1 f117q.sanityn68 -> 0 f117q.sanityn69 -> 0 f117q.sanityn70 -> 0 f117q.sanityn71 -> 0 f117q.sanityn72 -> 0 f117q.sanityn73 -> 0 f117q.sanityn74 -> 0 f117q.sanityn75 -> 0 f117q.sanityn76 -> 1 f117q.sanityn77 -> 0 f117q.sanityn78 -> 1 f117q.sanityn79 -> 0 f117q.sanityn80 -> 1 f117q.sanityn81 -> 1 f117q.sanityn82 -> 0 f117q.sanityn83 -> 0 f117q.sanityn84 -> 1 f117q.sanityn85 -> 0 f117q.sanityn86 -> 1 f117q.sanityn87 -> 1 f117q.sanityn88 -> 1 f117q.sanityn89 -> 0 f117q.sanityn90 -> 1 f117q.sanityn91 -> 0 f117q.sanityn92 -> 0 f117q.sanityn93 -> 1 f117q.sanityn94 -> 0 f117q.sanityn95 -> 1 f117q.sanityn96 -> 0 f117q.sanityn97 -> 1 f117q.sanityn98 -> 0 f117q.sanityn99 -> 0 - unlinked 0 (time 1766396649 ; total 0 ; last 0) total: 100 unlinks in 1 seconds: 100.000000 unlinks/second f117q.sanityn0 f117q.sanityn1 f117q.sanityn10 f117q.sanityn11 f117q.sanityn12 f117q.sanityn13 f117q.sanityn14 f117q.sanityn15 f117q.sanityn16 f117q.sanityn17 f117q.sanityn18 f117q.sanityn19 f117q.sanityn2 f117q.sanityn20 f117q.sanityn21 f117q.sanityn22 f117q.sanityn23 f117q.sanityn24 f117q.sanityn25 f117q.sanityn26 f117q.sanityn27 f117q.sanityn28 f117q.sanityn29 f117q.sanityn3 f117q.sanityn30 f117q.sanityn31 f117q.sanityn32 f117q.sanityn33 f117q.sanityn34 f117q.sanityn35 f117q.sanityn36 f117q.sanityn37 f117q.sanityn38 f117q.sanityn39 f117q.sanityn4 f117q.sanityn40 f117q.sanityn41 f117q.sanityn42 f117q.sanityn43 f117q.sanityn44 f117q.sanityn45 f117q.sanityn46 f117q.sanityn47 f117q.sanityn48 f117q.sanityn49 f117q.sanityn5 f117q.sanityn50 f117q.sanityn51 f117q.sanityn52 f117q.sanityn53 f117q.sanityn54 f117q.sanityn55 f117q.sanityn56 f117q.sanityn57 f117q.sanityn58 f117q.sanityn59 f117q.sanityn6 f117q.sanityn60 f117q.sanityn61 f117q.sanityn62 f117q.sanityn63 f117q.sanityn64 f117q.sanityn65 f117q.sanityn66 f117q.sanityn67 f117q.sanityn68 f117q.sanityn69 f117q.sanityn7 f117q.sanityn70 f117q.sanityn71 f117q.sanityn72 f117q.sanityn73 f117q.sanityn74 f117q.sanityn75 f117q.sanityn76 f117q.sanityn77 f117q.sanityn78 f117q.sanityn79 f117q.sanityn8 f117q.sanityn80 f117q.sanityn81 f117q.sanityn82 f117q.sanityn83 f117q.sanityn84 f117q.sanityn85 f117q.sanityn86 f117q.sanityn87 f117q.sanityn88 f117q.sanityn89 f117q.sanityn9 f117q.sanityn90 f117q.sanityn91 f117q.sanityn92 f117q.sanityn93 f117q.sanityn94 f117q.sanityn95 f117q.sanityn96 f117q.sanityn97 f117q.sanityn98 f117q.sanityn99 f117q.sanityn10 f117q.sanityn11 f117q.sanityn12 f117q.sanityn13 f117q.sanityn14 f117q.sanityn15 f117q.sanityn16 f117q.sanityn17 f117q.sanityn18 f117q.sanityn19 f117q.sanityn20 f117q.sanityn21 f117q.sanityn22 f117q.sanityn23 f117q.sanityn24 f117q.sanityn25 f117q.sanityn26 f117q.sanityn27 f117q.sanityn28 f117q.sanityn29 f117q.sanityn30 f117q.sanityn31 f117q.sanityn32 f117q.sanityn33 f117q.sanityn34 f117q.sanityn35 f117q.sanityn36 f117q.sanityn37 f117q.sanityn38 f117q.sanityn39 f117q.sanityn40 f117q.sanityn41 f117q.sanityn42 f117q.sanityn43 f117q.sanityn44 f117q.sanityn45 f117q.sanityn46 f117q.sanityn47 f117q.sanityn48 f117q.sanityn49 f117q.sanityn50 f117q.sanityn51 f117q.sanityn52 f117q.sanityn53 f117q.sanityn54 f117q.sanityn55 f117q.sanityn56 f117q.sanityn57 f117q.sanityn58 f117q.sanityn59 f117q.sanityn60 f117q.sanityn61 f117q.sanityn62 f117q.sanityn63 f117q.sanityn64 f117q.sanityn65 f117q.sanityn66 f117q.sanityn67 f117q.sanityn68 f117q.sanityn69 f117q.sanityn70 f117q.sanityn71 f117q.sanityn72 f117q.sanityn73 f117q.sanityn74 f117q.sanityn75 f117q.sanityn76 f117q.sanityn77 f117q.sanityn78 f117q.sanityn79 f117q.sanityn80 f117q.sanityn81 f117q.sanityn82 f117q.sanityn83 f117q.sanityn84 f117q.sanityn85 f117q.sanityn86 f117q.sanityn87 f117q.sanityn88 f117q.sanityn89 f117q.sanityn90 f117q.sanityn91 f117q.sanityn92 f117q.sanityn93 f117q.sanityn94 f117q.sanityn95 f117q.sanityn96 f117q.sanityn97 f117q.sanityn98 f117q.sanityn99 mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117q (20s) == sanityn test 117r: Verify the deleting a striped directory on Trash Can ========================================================== 04:44:21 (1766396661) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain #0 -> 0x200000bd0:0x823:0x0 #1 -> 0x240000404:0x79a:0x0 Creating 50 files under striped directory: /mnt/lustre/d117r.sanityn/striped_dir total: 50 open/close in 0.60 seconds: 83.96 ops/second f117r.sanityn0 -> 0 f117r.sanityn1 -> 0 f117r.sanityn2 -> 0 f117r.sanityn3 -> 0 f117r.sanityn4 -> 0 f117r.sanityn5 -> 0 f117r.sanityn6 -> 0 f117r.sanityn7 -> 0 f117r.sanityn8 -> 0 f117r.sanityn9 -> 0 f117r.sanityn10 -> 1 f117r.sanityn11 -> 1 f117r.sanityn12 -> 1 f117r.sanityn13 -> 1 f117r.sanityn14 -> 1 f117r.sanityn15 -> 0 f117r.sanityn16 -> 1 f117r.sanityn17 -> 1 f117r.sanityn18 -> 0 f117r.sanityn19 -> 1 f117r.sanityn20 -> 0 f117r.sanityn21 -> 0 f117r.sanityn22 -> 0 f117r.sanityn23 -> 0 f117r.sanityn24 -> 0 f117r.sanityn25 -> 1 f117r.sanityn26 -> 0 f117r.sanityn27 -> 1 f117r.sanityn28 -> 1 f117r.sanityn29 -> 1 f117r.sanityn30 -> 1 f117r.sanityn31 -> 0 f117r.sanityn32 -> 0 f117r.sanityn33 -> 0 f117r.sanityn34 -> 1 f117r.sanityn35 -> 0 f117r.sanityn36 -> 0 f117r.sanityn37 -> 0 f117r.sanityn38 -> 0 f117r.sanityn39 -> 1 f117r.sanityn40 -> 0 f117r.sanityn41 -> 1 f117r.sanityn42 -> 0 f117r.sanityn43 -> 1 f117r.sanityn44 -> 0 f117r.sanityn45 -> 1 f117r.sanityn46 -> 0 f117r.sanityn47 -> 0 f117r.sanityn48 -> 1 f117r.sanityn49 -> 0 - unlinked 0 (time 1766396667 ; total 0 ; last 0) total: 50 unlinks in 1 seconds: 50.000000 unlinks/second Check files in Trash Can: f117r.sanityn0 f117r.sanityn1 f117r.sanityn10 f117r.sanityn11 f117r.sanityn12 f117r.sanityn13 f117r.sanityn14 f117r.sanityn15 f117r.sanityn16 f117r.sanityn17 f117r.sanityn18 f117r.sanityn19 f117r.sanityn2 f117r.sanityn20 f117r.sanityn21 f117r.sanityn22 f117r.sanityn23 f117r.sanityn24 f117r.sanityn25 f117r.sanityn26 f117r.sanityn27 f117r.sanityn28 f117r.sanityn29 f117r.sanityn3 f117r.sanityn30 f117r.sanityn31 f117r.sanityn32 f117r.sanityn33 f117r.sanityn34 f117r.sanityn35 f117r.sanityn36 f117r.sanityn37 f117r.sanityn38 f117r.sanityn39 f117r.sanityn4 f117r.sanityn40 f117r.sanityn41 f117r.sanityn42 f117r.sanityn43 f117r.sanityn44 f117r.sanityn45 f117r.sanityn46 f117r.sanityn47 f117r.sanityn48 f117r.sanityn49 f117r.sanityn5 f117r.sanityn6 f117r.sanityn7 f117r.sanityn8 f117r.sanityn9 f117r.sanityn10 f117r.sanityn11 f117r.sanityn12 f117r.sanityn13 f117r.sanityn14 f117r.sanityn15 f117r.sanityn16 f117r.sanityn17 f117r.sanityn18 f117r.sanityn19 f117r.sanityn20 f117r.sanityn21 f117r.sanityn22 f117r.sanityn23 f117r.sanityn24 f117r.sanityn25 f117r.sanityn26 f117r.sanityn27 f117r.sanityn28 f117r.sanityn29 f117r.sanityn30 f117r.sanityn31 f117r.sanityn32 f117r.sanityn33 f117r.sanityn34 f117r.sanityn35 f117r.sanityn36 f117r.sanityn37 f117r.sanityn38 f117r.sanityn39 f117r.sanityn40 f117r.sanityn41 f117r.sanityn42 f117r.sanityn43 f117r.sanityn44 f117r.sanityn45 f117r.sanityn46 f117r.sanityn47 f117r.sanityn48 f117r.sanityn49 FID for /mnt/lustre/d117r.sanityn/striped_dir [0x200000bd2:0xc72:0x0] pfid[0]=0x200000bd0:0x823:0x0 [0x200000400:0x2:0x0] [0x200000bd0:0x824:0x0] pfid[1]=0x240000404:0x79a:0x0 [0x240000400:0x2:0x0] [0x240000400:0xe3b:0x0] Obtain LMVEA for virtual dir /mnt/lustre/d117r.sanityn/striped_dir/.Trash: lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x200000bd0:0x824:0x0] 1 [0x240000400:0xe3b:0x0] #0 -> name: 0x200000bd0:0x823:0x0 fid: 0x200000bd0:0x824:0x0 #1 -> name: 0x240000404:0x79a:0x0 fid: 0x240000400:0xe3b:0x0 /mnt/lustre/.lustre/trash: MDT0000 MDT0001 /mnt/lustre/.lustre/trash/MDT0000: 0 0x200000bd0:0x821:0x0 0x200000bd2:0xbcc:0x0 0x200000bd2:0xbd2:0x0 0x200000bd2:0xbe3:0x0 0x200000bd2:0xbe7:0x0 0x200000bd2:0xbec:0x0 0x200000bd2:0xbf3:0x0 0x200000bd2:0xc06:0x0 0x200000bd2:0xc0c:0x0 0x200000bd2:0xc12:0x0 0x200000bd2:0xc1c:0x0 0x200000bd2:0xc1f:0x0 0x200000bd2:0xc25:0x0 0x200000bd2:0xc28:0x0 0x200000bd2:0xc2e:0x0 0x200000bd2:0xc33:0x0 0x200000bd2:0xc37:0x0 0x200000bd2:0xc39:0x0 500 /mnt/lustre/.lustre/trash/MDT0000/0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd0:0x821:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbcc:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbd2:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe7:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbec:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbf3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc06:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc0c:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc12:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc1c:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc1f:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc25:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc28:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc2e:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc33:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc37:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc39:0x0: /mnt/lustre/.lustre/trash/MDT0000/500: /mnt/lustre/.lustre/trash/MDT0001: 0x240000404:0x799:0x0 0x240000405:0x41:0x0 0x240000405:0xc:0x0 /mnt/lustre/.lustre/trash/MDT0001/0x240000404:0x799:0x0: /mnt/lustre/.lustre/trash/MDT0001/0x240000405:0x41:0x0: striped_dir /mnt/lustre/.lustre/trash/MDT0001/0x240000405:0x41:0x0/striped_dir: f117r.sanityn0 f117r.sanityn1 f117r.sanityn10 f117r.sanityn11 f117r.sanityn12 f117r.sanityn13 f117r.sanityn14 f117r.sanityn15 f117r.sanityn16 f117r.sanityn17 f117r.sanityn18 f117r.sanityn19 f117r.sanityn2 f117r.sanityn20 f117r.sanityn21 f117r.sanityn22 f117r.sanityn23 f117r.sanityn24 f117r.sanityn25 f117r.sanityn26 f117r.sanityn27 f117r.sanityn28 f117r.sanityn29 f117r.sanityn3 f117r.sanityn30 f117r.sanityn31 f117r.sanityn32 f117r.sanityn33 f117r.sanityn34 f117r.sanityn35 f117r.sanityn36 f117r.sanityn37 f117r.sanityn38 f117r.sanityn39 f117r.sanityn4 f117r.sanityn40 f117r.sanityn41 f117r.sanityn42 f117r.sanityn43 f117r.sanityn44 f117r.sanityn45 f117r.sanityn46 f117r.sanityn47 f117r.sanityn48 f117r.sanityn49 f117r.sanityn5 f117r.sanityn6 f117r.sanityn7 f117r.sanityn8 f117r.sanityn9 file.1 file.2 /mnt/lustre/.lustre/trash/MDT0001/0x240000405:0xc:0x0: Obtain LMVEA for striped dir /mnt/lustre/d117r.sanityn/striped_dir: lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x200000bd0:0x824:0x0] 1 [0x240000400:0xe3b:0x0] [0x200000bd0:0x826:0x0] #0 -> name: 0x200000bd0:0x823:0x0 fid: 0x200000bd0:0x824:0x0 #1 -> name: 0x240000404:0x79a:0x0 fid: 0x240000400:0xe3b:0x0 lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x200000bd0:0x824:0x0] 1 [0x240000400:0xe3b:0x0] f117r.sanityn0 f117r.sanityn1 f117r.sanityn10 f117r.sanityn11 f117r.sanityn12 f117r.sanityn13 f117r.sanityn14 f117r.sanityn15 f117r.sanityn16 f117r.sanityn17 f117r.sanityn18 f117r.sanityn19 f117r.sanityn2 f117r.sanityn20 f117r.sanityn21 f117r.sanityn22 f117r.sanityn23 f117r.sanityn24 f117r.sanityn25 f117r.sanityn26 f117r.sanityn27 f117r.sanityn28 f117r.sanityn29 f117r.sanityn3 f117r.sanityn30 f117r.sanityn31 f117r.sanityn32 f117r.sanityn33 f117r.sanityn34 f117r.sanityn35 f117r.sanityn36 f117r.sanityn37 f117r.sanityn38 f117r.sanityn39 f117r.sanityn4 f117r.sanityn40 f117r.sanityn41 f117r.sanityn42 f117r.sanityn43 f117r.sanityn44 f117r.sanityn45 f117r.sanityn46 f117r.sanityn47 f117r.sanityn48 f117r.sanityn49 f117r.sanityn5 f117r.sanityn6 f117r.sanityn7 f117r.sanityn8 f117r.sanityn9 file.1 file.2 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn0, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396665 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn1, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396665 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn10, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396666 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn11, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396666 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn12, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396666 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn13, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396666 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn14, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396666 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn15, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396666 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn16, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396666 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn17, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396666 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn18, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396666 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn19, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396666 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn2, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396665 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn20, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396666 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn21, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396666 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn22, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396666 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn23, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396666 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn24, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396666 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn25, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396666 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn26, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396666 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn27, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396666 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn28, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396666 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn29, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396666 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn3, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396665 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn30, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396666 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn31, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396666 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn32, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396666 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn33, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396666 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn34, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396666 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn35, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396666 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn36, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396666 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn37, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396666 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn38, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396666 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn39, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396666 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn4, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396665 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn40, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396666 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn41, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396666 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn42, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396666 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn43, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396666 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn44, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396666 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn45, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396666 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn46, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396666 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn47, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396666 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn48, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396666 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn49, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396666 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn5, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396665 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn6, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396665 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn7, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396665 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn8, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396665 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn9, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396665 file: /mnt/lustre/d117r.sanityn/striped_dir/file.1, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396665 file: /mnt/lustre/d117r.sanityn/striped_dir/file.2, flags: 0X80000, uid: 0 gid: 0 projid: 0 timestamp: 1766396665 mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117r (20s) == sanityn test 117s: Create trash shard if it does not exist ========================================================== 04:44:41 (1766396681) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain /mnt/lustre/.lustre/trash: MDT0000 MDT0001 /mnt/lustre/.lustre/trash/MDT0000: 0 0x200000bd0:0x821:0x0 0x200000bd2:0xbcc:0x0 0x200000bd2:0xbd2:0x0 0x200000bd2:0xbe3:0x0 0x200000bd2:0xbe7:0x0 0x200000bd2:0xbec:0x0 0x200000bd2:0xbf3:0x0 0x200000bd2:0xc06:0x0 0x200000bd2:0xc0c:0x0 0x200000bd2:0xc12:0x0 0x200000bd2:0xc1c:0x0 0x200000bd2:0xc1f:0x0 0x200000bd2:0xc25:0x0 0x200000bd2:0xc28:0x0 0x200000bd2:0xc2e:0x0 0x200000bd2:0xc33:0x0 0x200000bd2:0xc37:0x0 0x200000bd2:0xc39:0x0 500 /mnt/lustre/.lustre/trash/MDT0000/0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd0:0x821:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbcc:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbd2:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe7:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbec:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbf3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc06:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc0c:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc12:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc1c:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc1f:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc25:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc28:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc2e:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc33:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc37:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc39:0x0: /mnt/lustre/.lustre/trash/MDT0000/500: /mnt/lustre/.lustre/trash/MDT0001: 0x240000404:0x799:0x0 0x240000404:0x79b:0x0 0x240000405:0x41:0x0 0x240000405:0xc:0x0 /mnt/lustre/.lustre/trash/MDT0001/0x240000404:0x799:0x0: /mnt/lustre/.lustre/trash/MDT0001/0x240000404:0x79b:0x0: file.1 /mnt/lustre/.lustre/trash/MDT0001/0x240000405:0x41:0x0: /mnt/lustre/.lustre/trash/MDT0001/0x240000405:0xc:0x0: lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x200000bd0:0x829:0x0] 1 [0x240000404:0x79b:0x0] mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117s (9s) == sanityn test 117t: UID/GID/ProjID for files in Trash Can ========================================================== 04:44:50 (1766396690) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain file: /mnt/lustre/d117t.sanityn/.Trash/tfile1, flags: 0X80002 unrm, uid: 0 gid: 0 projid: 0 timestamp: 1766396694 File: /mnt/lustre/d117t.sanityn/.Trash/tfile1 Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238843714710 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 04:44:53.000000000 -0500 Modify: 2025-12-22 04:44:53.000000000 -0500 Change: 2025-12-22 04:44:53.000000000 -0500 Birth: 2025-12-22 04:44:54.000000000 -0500 trash_can_uid=4294967294 trash_can_gid=4294967294 trash_can_projid=4294967294 === Checking ugp ID on MDT === oleg148-server: debugfs 1.47.3-wc2 (11-Nov-2025) Inode: 519296 Type: regular Mode: 0644 Flags: 0x80002 Generation: 927412597 Version: 0x00000002:00019d6c User: -2 Group: -2 Project: -2 Size: 0 File ACL: 0 Links: 1 Blockcount: 0 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x69491315:00000000 -- Mon Dec 22 04:44:53 2025 atime: 0x69491315:00000000 -- Mon Dec 22 04:44:53 2025 mtime: 0x69491315:00000000 -- Mon Dec 22 04:44:53 2025 crtime: 0x69491316:62b48e00 -- Mon Dec 22 04:44:54 2025 Size of extra inode fields: 32 Extended attributes: lma: fid=[0x200000bd2:0xc96:0x0] compat=0 incompat=0 user.job (5) = "lfs.0" linkea: idx=0 parent=[0x200000bd0:0x82d:0x0] name='tfile1' trusted.lov (56) = d0 0b d1 0b 01 00 00 00 96 0c 00 00 00 00 00 00 d2 0b 00 00 02 00 00 00 00 00 40 00 01 00 00 00 01 04 00 80 02 00 00 00 96 19 00 00 00 00 00 00 00 00 00 00 00 00 00 00 trusted.som (24) = 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 trusted.unrm (24) = 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 16 13 49 69 00 00 00 00 EXTENTS: oleg148-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg148-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg148-server: debugfs 1.47.3-wc2 (11-Nov-2025) === Checking ugp ID on OST === oleg148-server: debugfs 1.47.3-wc2 (11-Nov-2025) Inode: 3497 Type: regular Mode: 0666 Flags: 0x80000 Generation: 1321432932 Version: 0x00000000:00000000 User: -2 Group: -2 Project: -2 Size: 0 File ACL: 0 Links: 1 Blockcount: 0 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x00000000:00000000 -- Wed Dec 31 19:00:00 1969 atime: 0x00000000:00000000 -- Wed Dec 31 19:00:00 1969 mtime: 0x00000000:00000000 -- Wed Dec 31 19:00:00 1969 crtime: 0x694912e6:e9696b00 -- Mon Dec 22 04:44:06 2025 Size of extra inode fields: 32 Extended attributes: lma: fid=[0x280000401:0x1996:0x0] compat=8 incompat=0 EXTENTS: oleg148-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg148-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg148-server: debugfs 1.47.3-wc2 (11-Nov-2025) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d117t.sanityn/tfile2] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [mkdir] [/mnt/lustre/d117t.sanityn/tdir2] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [unlink] [/mnt/lustre/d117t.sanityn/tfile2] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [rmdir] [/mnt/lustre/d117t.sanityn/tdir2] file: /mnt/lustre/d117t.sanityn/.Trash/tfile2, flags: 0X80002 unrm, uid: 500 gid: 500 projid: 0 timestamp: 1766396700 File: /mnt/lustre/d117t.sanityn/.Trash/tfile2 Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238843714711 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 500/sanityusr) Gid: ( 500/sanityusr) Access: 2025-12-22 04:44:59.000000000 -0500 Modify: 2025-12-22 04:44:59.000000000 -0500 Change: 2025-12-22 04:44:59.000000000 -0500 Birth: 2025-12-22 04:45:00.000000000 -0500 using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg148-client.virtnet /mnt/lustre (opts:) Stopping client oleg148-client.virtnet /mnt/lustre opts: Stopping clients: oleg148-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg148-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg148-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg148-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg148-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg148-server tune2fs 1.47.3-wc2 (11-Nov-2025) tune2fs 1.47.3-wc2 (11-Nov-2025) tune2fs 1.47.3-wc2 (11-Nov-2025) tune2fs 1.47.3-wc2 (11-Nov-2025) sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg148-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.48,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg148-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.48,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg148-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.48,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/58764/testresults/sanityn-ldiskfs-DNE-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.48,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg148-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Starting client oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Started clients oleg148-client.virtnet: 192.168.201.148@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 Starting client oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 Started clients oleg148-client.virtnet: 192.168.201.148@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9e1b82f89800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9e1b87e19000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e1b82f89800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e1b87e19000.idle_timeout=debug disable quota as required mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117t (138s) == sanityn test 117u: statfs() call should not account the space in Trash Can ========================================================== 04:47:08 (1766396828) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain mdd.lustre-MDT0000.trash_can_projid=1000 mdd.lustre-MDT0001.trash_can_projid=1000 llite.lustre-ffff9e1b82f89800.trash_can_projid=1000 llite.lustre-ffff9e1b87e19000.trash_can_projid=1000 total=7666232 used=86716 avail=7127324 free=7579516 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.148@tcp:/lustre 7666232 86716 7127324 2% /mnt/lustre 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.07097 s, 34.1 MB/s new total=7666232 used=0 avail=7666232 free=7666232 Filesystem 1024-blocks Used Available Capacity Mounted on 192.168.201.148@tcp:/lustre 7666232 0 7666232 0% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 6380 1264924 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 5164 1266140 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37436 3569584 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 151680 3455340 5% /mnt/lustre[OST:1] filesystem_summary: 7666232 189116 7024924 3% /mnt/lustre /mnt/lustre/.lustre/trash: MDT0000 MDT0001 /mnt/lustre/.lustre/trash/MDT0000: 0 0x200000bd0:0x821:0x0 0x200000bd2:0xbcc:0x0 0x200000bd2:0xbd2:0x0 0x200000bd2:0xbe3:0x0 0x200000bd2:0xbe7:0x0 0x200000bd2:0xbec:0x0 0x200000bd2:0xbf3:0x0 0x200000bd2:0xc06:0x0 0x200000bd2:0xc0c:0x0 0x200000bd2:0xc12:0x0 0x200000bd2:0xc1c:0x0 0x200000bd2:0xc1f:0x0 0x200000bd2:0xc25:0x0 0x200000bd2:0xc28:0x0 0x200000bd2:0xc2e:0x0 0x200000bd2:0xc33:0x0 0x200000bd2:0xc37:0x0 0x200000bd2:0xc39:0x0 0x200000bd2:0xc93:0x0 0x200000bd2:0xc95:0x0 0x2000013a1:0x3:0x0 500 /mnt/lustre/.lustre/trash/MDT0000/0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd0:0x821:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbcc:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbd2:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe7:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbec:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbf3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc06:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc0c:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc12:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc1c:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc1f:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc25:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc28:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc2e:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc33:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc37:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc39:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc93:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc95:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x2000013a1:0x3:0x0: f117u.sanityn /mnt/lustre/.lustre/trash/MDT0000/500: /mnt/lustre/.lustre/trash/MDT0001: 0x240000404:0x799:0x0 0x240000404:0x79b:0x0 0x240000405:0x41:0x0 0x240000405:0xc:0x0 /mnt/lustre/.lustre/trash/MDT0001/0x240000404:0x799:0x0: /mnt/lustre/.lustre/trash/MDT0001/0x240000404:0x79b:0x0: /mnt/lustre/.lustre/trash/MDT0001/0x240000405:0x41:0x0: /mnt/lustre/.lustre/trash/MDT0001/0x240000405:0xc:0x0: PROJID: 1000 Quota Info: Disk quotas for prj (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 102400 0 0 - 1 0 0 - lustre-MDT0000 0 - 0 - 1 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 102400 - 0 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for usr green (uid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 total=7666232 used=189116 avail=7024924 free=7477116 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 6380 1264924 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 5164 1266140 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37436 3569584 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 151680 3455340 5% /mnt/lustre[OST:1] filesystem_summary: 7666232 189116 7024924 3% /mnt/lustre new total=7666232 used=37436 avail=7402700 free=7628796 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 6380 1264924 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 5164 1266140 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37436 3569584 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 0 3833116 0% /mnt/lustre[OST:1] filesystem_summary: 7666232 37436 7402700 1% /mnt/lustre llite.lustre-ffff9e1b82f89800.trash_can_projid=4294967294 llite.lustre-ffff9e1b87e19000.trash_can_projid=4294967294 mdd.lustre-MDT0000.trash_can_uid=4294967294 mdd.lustre-MDT0001.trash_can_uid=4294967294 mdd.lustre-MDT0000.trash_can_gid=4294967294 mdd.lustre-MDT0001.trash_can_gid=4294967294 mdd.lustre-MDT0000.trash_can_projid=4294967294 mdd.lustre-MDT0001.trash_can_projid=4294967294 mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117u (27s) == sanityn test 117ub: statfs() account the inode usage in Trash Can ========================================================== 04:47:35 (1766396855) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1024000 12558 1011442 2% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1024000 9307 1014693 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 262144 3685 258459 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 262144 3675 258469 2% /mnt/lustre[OST:1] filesystem_summary: 538793 21865 516928 5% /mnt/lustre - open/close 939 (time 1766396874.46 total 10.01 last 93.82) total: 1000 open/close in 10.91 seconds: 91.64 ops/second - unlinked 0 (time 1766396878 ; total 0 ; last 0) total: 1000 unlinks in 5 seconds: 200.000000 unlinks/second === w/o Trash Can account === lustre-MDT0000_UUID 1024000 13560 1010440 2% /mnt/lustre[MDT:0] === w/ Trash Can account === lustre-MDT0000_UUID 1024000 12560 1011440 2% /mnt/lustre[MDT:0] tcu_ffree: 1011440 ffree: 1010440 mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117ub (51s) == sanityn test 117v: Test for Trash Can UID/GID/PROJID setting ========================================================== 04:48:26 (1766396906) mdd.lustre-MDT0000.trash_can_uid=10 mdd.lustre-MDT0001.trash_can_uid=10 mdd.lustre-MDT0000.trash_can_gid=10 mdd.lustre-MDT0001.trash_can_gid=10 mdd.lustre-MDT0000.trash_can_projid=10 mdd.lustre-MDT0001.trash_can_projid=10 llite.lustre-ffff9e1b82f89800.trash_can_projid=10 llite.lustre-ffff9e1b87e19000.trash_can_projid=10 mdd.lustre-MDT0000.trash_can_uid=-1 mdd.lustre-MDT0001.trash_can_uid=-1 mdd.lustre-MDT0000.trash_can_gid=-1 mdd.lustre-MDT0001.trash_can_gid=-1 mdd.lustre-MDT0000.trash_can_projid=-1 mdd.lustre-MDT0001.trash_can_projid=-1 llite.lustre-ffff9e1b82f89800.trash_can_projid=-1 llite.lustre-ffff9e1b87e19000.trash_can_projid=-1 mdd.lustre-MDT0000.trash_can_uid=-2 mdd.lustre-MDT0001.trash_can_uid=-2 mdd.lustre-MDT0000.trash_can_gid=-2 mdd.lustre-MDT0001.trash_can_gid=-2 mdd.lustre-MDT0000.trash_can_projid=-2 mdd.lustre-MDT0001.trash_can_projid=-2 llite.lustre-ffff9e1b82f89800.trash_can_projid=-2 llite.lustre-ffff9e1b87e19000.trash_can_projid=-2 mdd.lustre-MDT0000.trash_can_uid=original mdd.lustre-MDT0001.trash_can_uid=original mdd.lustre-MDT0000.trash_can_gid=original mdd.lustre-MDT0001.trash_can_gid=original mdd.lustre-MDT0000.trash_can_projid=original mdd.lustre-MDT0001.trash_can_projid=original llite.lustre-ffff9e1b82f89800.trash_can_projid=original llite.lustre-ffff9e1b87e19000.trash_can_projid=original mdd.lustre-MDT0000.trash_can_uid=default mdd.lustre-MDT0001.trash_can_uid=default mdd.lustre-MDT0000.trash_can_gid=default mdd.lustre-MDT0001.trash_can_gid=default mdd.lustre-MDT0000.trash_can_projid=default mdd.lustre-MDT0001.trash_can_projid=default llite.lustre-ffff9e1b82f89800.trash_can_projid=default llite.lustre-ffff9e1b87e19000.trash_can_projid=default llite.lustre-ffff9e1b82f89800.trash_can_projid=4294967294 llite.lustre-ffff9e1b87e19000.trash_can_projid=4294967294 mdd.lustre-MDT0000.trash_can_uid=4294967294 mdd.lustre-MDT0001.trash_can_uid=4294967294 mdd.lustre-MDT0000.trash_can_gid=4294967294 mdd.lustre-MDT0001.trash_can_gid=4294967294 mdd.lustre-MDT0000.trash_can_projid=4294967294 mdd.lustre-MDT0001.trash_can_projid=4294967294 PASS 117v (14s) == sanityn test 117w: Test Trash Can with original UID/GID/PROJID ========================================================== 04:48:40 (1766396920) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain mdd.lustre-MDT0000.trash_can_uid=original mdd.lustre-MDT0001.trash_can_uid=original mdd.lustre-MDT0000.trash_can_gid=original mdd.lustre-MDT0001.trash_can_gid=original mdd.lustre-MDT0000.trash_can_projid=original mdd.lustre-MDT0001.trash_can_projid=original 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.14512 s, 48.9 MB/s /mnt/lustre/.lustre/trash: MDT0000 MDT0001 /mnt/lustre/.lustre/trash/MDT0000: 0 0x200000bd0:0x821:0x0 0x200000bd2:0xbcc:0x0 0x200000bd2:0xbd2:0x0 0x200000bd2:0xbe3:0x0 0x200000bd2:0xbe7:0x0 0x200000bd2:0xbec:0x0 0x200000bd2:0xbf3:0x0 0x200000bd2:0xc06:0x0 0x200000bd2:0xc0c:0x0 0x200000bd2:0xc12:0x0 0x200000bd2:0xc1c:0x0 0x200000bd2:0xc1f:0x0 0x200000bd2:0xc25:0x0 0x200000bd2:0xc28:0x0 0x200000bd2:0xc2e:0x0 0x200000bd2:0xc33:0x0 0x200000bd2:0xc37:0x0 0x200000bd2:0xc39:0x0 0x200000bd2:0xc93:0x0 0x200000bd2:0xc95:0x0 0x2000013a1:0x3:0x0 0x2000013a1:0x3ee:0x0 0x2000013a1:0x5:0x0 500 /mnt/lustre/.lustre/trash/MDT0000/0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd0:0x821:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbcc:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbd2:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe7:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbec:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbf3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc06:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc0c:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc12:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc1c:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc1f:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc25:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc28:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc2e:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc33:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc37:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc39:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc93:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc95:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x2000013a1:0x3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x2000013a1:0x3ee:0x0: tdir.1 tfile.1 /mnt/lustre/.lustre/trash/MDT0000/0x2000013a1:0x3ee:0x0/tdir.1: /mnt/lustre/.lustre/trash/MDT0000/0x2000013a1:0x5:0x0: /mnt/lustre/.lustre/trash/MDT0000/500: /mnt/lustre/.lustre/trash/MDT0001: 0x240000404:0x799:0x0 0x240000404:0x79b:0x0 0x240000405:0x41:0x0 0x240000405:0xc:0x0 /mnt/lustre/.lustre/trash/MDT0001/0x240000404:0x799:0x0: /mnt/lustre/.lustre/trash/MDT0001/0x240000404:0x79b:0x0: /mnt/lustre/.lustre/trash/MDT0001/0x240000405:0x41:0x0: /mnt/lustre/.lustre/trash/MDT0001/0x240000405:0xc:0x0: oleg148-server: debugfs 1.47.3-wc2 (11-Nov-2025) Inode: 519303 Type: regular Mode: 0644 Flags: 0x80002 Generation: 505123478 Version: 0x00000003:000010bb User: 0 Group: 0 Project: 0 Size: 0 File ACL: 0 Links: 1 Blockcount: 0 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x69491401:00000000 -- Mon Dec 22 04:48:49 2025 atime: 0x694913ff:00000000 -- Mon Dec 22 04:48:47 2025 mtime: 0x69491401:00000000 -- Mon Dec 22 04:48:49 2025 crtime: 0x694913ff:7b062400 -- Mon Dec 22 04:48:47 2025 Size of extra inode fields: 32 Extended attributes: lma: fid=[0x2000013a1:0x3f0:0x0] compat=0 incompat=0 trusted.lov (56) = d0 0b d1 0b 01 00 00 00 f0 03 00 00 00 00 00 00 a1 13 00 00 02 00 00 00 00 00 40 00 01 00 00 00 01 04 00 80 02 00 00 00 b7 1b 00 00 00 00 00 00 00 00 00 00 00 00 00 00 user.job (4) = "dd.0" linkea: idx=0 parent=[0x2000013a0:0x3:0x0] name='tfile.1' trusted.som (24) = 04 00 00 00 00 00 00 00 00 00 40 06 00 00 00 00 08 20 03 00 00 00 00 00 EXTENTS: oleg148-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg148-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg148-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg148-server: debugfs 1.47.3-wc2 (11-Nov-2025) Inode: 519302 Type: directory Mode: 0755 Flags: 0x2 Generation: 1387173793 Version: 0x00000003:000010ba User: 0 Group: 0 Project: 0 Size: 4096 File ACL: 0 Links: 4 Blockcount: 8 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x69491401:00000000 -- Mon Dec 22 04:48:49 2025 atime: 0x694913ff:00000000 -- Mon Dec 22 04:48:47 2025 mtime: 0x694913ff:00000000 -- Mon Dec 22 04:48:47 2025 crtime: 0x694913ff:77359400 -- Mon Dec 22 04:48:47 2025 Size of extra inode fields: 32 Extended attributes: lma: fid=[0x2000013a1:0x3ef:0x0] compat=0 incompat=0 user.job (7) = "mkdir.0" linkea: idx=0 parent=[0x2000013a0:0x3:0x0] name='tdir.1' BLOCKS: (0):345718 TOTAL: 1 oleg148-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg148-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg148-server: debugfs 1.47.3-wc2 (11-Nov-2025) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [mkdir] [/mnt/lustre/d117w.sanityn/tdir.2] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d117w.sanityn/tfile.2] oleg148-server: debugfs 1.47.3-wc2 (11-Nov-2025) Inode: 519306 Type: regular Mode: 0644 Flags: 0x80002 Generation: 1284190775 Version: 0x00000003:000010f3 User: 500 Group: 500 Project: 0 Size: 0 File ACL: 0 Links: 1 Blockcount: 0 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x69491406:00000000 -- Mon Dec 22 04:48:54 2025 atime: 0x69491406:00000000 -- Mon Dec 22 04:48:54 2025 mtime: 0x69491406:00000000 -- Mon Dec 22 04:48:54 2025 crtime: 0x69491407:10ed7f00 -- Mon Dec 22 04:48:55 2025 Size of extra inode fields: 32 Extended attributes: lma: fid=[0x2000013a1:0x3f2:0x0] compat=0 incompat=0 trusted.lov (56) = d0 0b d1 0b 01 00 00 00 f2 03 00 00 00 00 00 00 a1 13 00 00 02 00 00 00 00 00 40 00 01 00 00 00 01 04 00 c0 02 00 00 00 77 1b 00 00 00 00 00 00 00 00 00 00 01 00 00 00 user.job (9) = "touch.500" linkea: idx=0 parent=[0x2000013a0:0x3:0x0] name='tfile.2' trusted.som (24) = 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 EXTENTS: oleg148-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg148-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg148-server: debugfs 1.47.3-wc2 (11-Nov-2025) Inode: 519305 Type: directory Mode: 0755 Flags: 0x2 Generation: 3255035755 Version: 0x00000003:000010f2 User: 500 Group: 500 Project: 0 Size: 4096 File ACL: 0 Links: 4 Blockcount: 8 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x69491406:00000000 -- Mon Dec 22 04:48:54 2025 atime: 0x69491406:00000000 -- Mon Dec 22 04:48:54 2025 mtime: 0x69491406:00000000 -- Mon Dec 22 04:48:54 2025 crtime: 0x69491407:0aba9500 -- Mon Dec 22 04:48:55 2025 Size of extra inode fields: 32 Extended attributes: lma: fid=[0x2000013a1:0x3f1:0x0] compat=0 incompat=0 user.job (9) = "mkdir.500" linkea: idx=0 parent=[0x2000013a0:0x3:0x0] name='tdir.2' BLOCKS: (0):345720 TOTAL: 1 oleg148-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg148-server: debugfs 1.47.3-wc2 (11-Nov-2025) mdd.lustre-MDT0000.trash_can_uid=4294967294 mdd.lustre-MDT0001.trash_can_uid=4294967294 mdd.lustre-MDT0000.trash_can_gid=4294967294 mdd.lustre-MDT0001.trash_can_gid=4294967294 mdd.lustre-MDT0000.trash_can_projid=4294967294 mdd.lustre-MDT0001.trash_can_projid=4294967294 mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117w (24s) == sanityn test 117x: Unrm Files from Trash Can should also restore UID/GID/PROJID ========================================================== 04:49:04 (1766396944) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/d117x.sanityn/.Trash/tdir1 trusted.unrm=0sAAAAAAAAAAAAAAAAAAAAABUUSWkAAAAA getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/d117x.sanityn/.Trash/tfile1 trusted.unrm=0sAAAAAAAAAAAAAAAAAAAAABUUSWkAAAAA /mnt/lustre/d117x.sanityn/tdir1: trusted.unrm: No such attribute /mnt/lustre/d117x.sanityn/tfile1: trusted.unrm: No such attribute running as uid/gid/euid/egid 500/500/500/500, groups: 500 [mkdir] [/mnt/lustre/d117x.sanityn/tdir2] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d117x.sanityn/tfile2] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [rmdir] [/mnt/lustre/d117x.sanityn/tdir2] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [unlink] [/mnt/lustre/d117x.sanityn/tfile2] getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/d117x.sanityn/.Trash/tdir2 trusted.unrm=0sAAAAAPQBAAD0AQAAAAAAABYUSWkAAAAA getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/d117x.sanityn/.Trash/tfile2 trusted.unrm=0sAAAAAPQBAAD0AQAAAAAAABYUSWkAAAAA /mnt/lustre/d117x.sanityn/tdir2: trusted.unrm: No such attribute /mnt/lustre/d117x.sanityn/tfile2: trusted.unrm: No such attribute mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117x (11s) == sanityn test 117ya: Test with symbol links with Trash Can ========================================================== 04:49:15 (1766396955) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain total 8 lrwxrwxrwx 1 root root 32 Dec 22 04:49 lnkd5 -> /mnt/lustre/d117ya.sanityn/tdir1 lrwxrwxrwx 1 root root 33 Dec 22 04:49 lnkf4 -> /mnt/lustre/d117ya.sanityn/tfile3 drwxr-xr-x 3 root root 4096 Dec 22 04:49 tdir1 drwxr-xr-x 2 root root 4096 Dec 22 04:49 tdir2 -rw-r--r-- 1 root root 0 Dec 22 04:49 tfile3 After unlink /mnt/lustre/d117ya.sanityn/lnkf4 and /mnt/lustre/d117ya.sanityn/lnkd5 /mnt/lustre/d117ya.sanityn/.Trash: total 4 lrwxrwxrwx 1 root root 32 Dec 22 04:49 lnkd5 -> /mnt/lustre/d117ya.sanityn/tdir1 lrwxrwxrwx 1 root root 33 Dec 22 04:49 lnkf4 -> /mnt/lustre/d117ya.sanityn/tfile3 d--------- 3 root root 4096 Dec 22 04:49 tdir1 -rw-r--r-- 1 root root 0 Dec 22 04:49 tfile3 /mnt/lustre/d117ya.sanityn/.Trash/tdir1: total 4 drwxr-xr-x 4 root root 4096 Dec 22 04:49 tdir13 -rw-r--r-- 1 root root 0 Dec 22 04:49 tfile11 -rw-r--r-- 1 root root 0 Dec 22 04:49 tfile12 /mnt/lustre/d117ya.sanityn/.Trash/tdir1/tdir13: total 0 file: /mnt/lustre/d117ya.sanityn/.Trash/lnkf4, flags: 0X2 unrm, uid: 0 gid: 0 projid: 0 timestamp: 1766396960 file: /mnt/lustre/d117ya.sanityn/.Trash/lnkd5, flags: 0X2 unrm, uid: 0 gid: 0 projid: 0 timestamp: 1766396960 Unrm stub file 'lnkd5' under directory '/mnt/lustre/d117ya.sanityn' Check if '/mnt/lustre/d117ya.sanityn/.Trash/lnkd5' is in Trash Can Path '/mnt/lustre/d117ya.sanityn/.Trash/lnkd5' is a symlink Handle symlink /mnt/lustre/d117ya.sanityn/.Trash/lnkd5 Unrm stub file 'tdir1' under directory '/mnt/lustre/d117ya.sanityn' Check if '/mnt/lustre/d117ya.sanityn/.Trash/tdir1' is in Trash Can Set flags 0X2/0 OK: rc=0 Handle dir 0:'tdir13' under directory '/mnt/lustre/d117ya.sanityn/tdir1' Handle special file 0:/mnt/lustre/d117ya.sanityn/tdir1/tdir13 Handle file 0:'tfile12' under directory '/mnt/lustre/d117ya.sanityn/tdir1' Handle special file 0:/mnt/lustre/d117ya.sanityn/tdir1/tfile12 Handle file 0:'tfile11' under directory '/mnt/lustre/d117ya.sanityn/tdir1' Handle special file 0:/mnt/lustre/d117ya.sanityn/tdir1/tfile11 Unrm stub file 'tfile3' under directory '/mnt/lustre/d117ya.sanityn' Check if '/mnt/lustre/d117ya.sanityn/.Trash/tfile3' is in Trash Can Set flags 0X80002/0X80000 OK: rc=0 Unrm stub file 'lnkf4' under directory '/mnt/lustre/d117ya.sanityn' Check if '/mnt/lustre/d117ya.sanityn/.Trash/lnkf4' is in Trash Can Path '/mnt/lustre/d117ya.sanityn/.Trash/lnkf4' is a symlink Handle symlink /mnt/lustre/d117ya.sanityn/.Trash/lnkf4 mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117ya (10s) == sanityn test 117yb: Compare the directories containing symlinks after unrm recover ========================================================== 04:49:25 (1766396965) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain Only in /etc/pam.d: password-auth Only in /etc/postfix: master.cf.proto Only in /etc: subgid Only in /etc: subuid- sanityn test_117yb: @@@@@@ FAIL: /etc and /mnt/lustre/d117yb.sanityn/etc diff Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7318:error() = /home/green/git/lustre-release/lustre/tests/sanityn.sh:8873:test_117yb() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7694:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7757:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7572:run_test() = /home/green/git/lustre-release/lustre/tests/sanityn.sh:8875:main() Dumping lctl log to /tmp/testlogs//sanityn.test_117yb.*.1766397047.log rsync: chown "/tmp/testlogs/.sanityn.test_117yb.debug_log.oleg148-server.1766397047.log.3IPIoO" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanityn.test_117yb.dmesg.oleg148-server.1766397047.log.A550FY" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid FAIL 117yb (86s) == sanityn test 117yc: Restoring UID/GID/PROJID for symlinks works ========================================================== 04:50:51 (1766397051) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg148-client.virtnet /mnt/lustre (opts:) Stopping client oleg148-client.virtnet /mnt/lustre opts: Stopping clients: oleg148-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg148-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg148-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg148-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg148-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg148-server tune2fs 1.47.3-wc2 (11-Nov-2025) tune2fs 1.47.3-wc2 (11-Nov-2025) tune2fs 1.47.3-wc2 (11-Nov-2025) tune2fs 1.47.3-wc2 (11-Nov-2025) sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg148-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.48,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg148-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.48,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg148-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.48,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/58764/testresults/sanityn-ldiskfs-DNE-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.48,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg148-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Starting client oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre Started clients oleg148-client.virtnet: 192.168.201.148@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 Starting client oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 Started clients oleg148-client.virtnet: 192.168.201.148@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9e1b87e1e800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9e1b8ac5a800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e1b87e1e800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e1b8ac5a800.idle_timeout=debug disable quota as required mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain running as uid/gid/euid/egid 500/500/500/500, groups: 500 [mkdir] [/mnt/lustre/d117yc.sanityn/tdir1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d117yc.sanityn/tfile1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [ln] [-s] [/mnt/lustre/d117yc.sanityn/tfile1] [/mnt/lustre/d117yc.sanityn/lnkf3] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [ln] [-s] [/mnt/lustre/d117yc.sanityn/tdir1] [/mnt/lustre/d117yc.sanityn/lnkd4] getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/d117yc.sanityn/.Trash/lnkf3 trusted.unrm=0sAAAAAPQBAAD0AQAAyAAAANIUSWkAAAAA getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/d117yc.sanityn/.Trash/lnkd4 trusted.unrm=0sAAAAAPQBAAD0AQAAAAAAANIUSWkAAAAA Unrm stub file 'lnkd4' under directory '/mnt/lustre/d117yc.sanityn' Check if '/mnt/lustre/d117yc.sanityn/.Trash/lnkd4' is in Trash Can Path '/mnt/lustre/d117yc.sanityn/.Trash/lnkd4' is a symlink Handle symlink /mnt/lustre/d117yc.sanityn/.Trash/lnkd4 Unrm stub file 'tfile1' under directory '/mnt/lustre/d117yc.sanityn' Check if '/mnt/lustre/d117yc.sanityn/.Trash/tfile1' is in Trash Can Set flags 0X80002/0X80000 OK: rc=0 Unrm stub file 'lnkf3' under directory '/mnt/lustre/d117yc.sanityn' Check if '/mnt/lustre/d117yc.sanityn/.Trash/lnkf3' is in Trash Can Path '/mnt/lustre/d117yc.sanityn/.Trash/lnkf3' is a symlink Handle symlink /mnt/lustre/d117yc.sanityn/.Trash/lnkf3 Unrm stub file 'tdir1' under directory '/mnt/lustre/d117yc.sanityn' Check if '/mnt/lustre/d117yc.sanityn/.Trash/tdir1' is in Trash Can Set flags 0X2/0 OK: rc=0 /mnt/lustre/d117yc.sanityn/lnkf3: trusted.unrm: No such attribute /mnt/lustre/d117yc.sanityn/lnkd4: trusted.unrm: No such attribute mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117yc (91s) == sanityn test 117za: Test Trash Can for UID Trash Can type with nodemap ========================================================== 04:52:22 (1766397142) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid Trash Can Type: uid uid /mnt/lustre/.lustre/trash: MDT0000 MDT0001 /mnt/lustre/.lustre/trash/MDT0000: 0 0x200000bd0:0x821:0x0 0x200000bd2:0xbcc:0x0 0x200000bd2:0xbd2:0x0 0x200000bd2:0xbe3:0x0 0x200000bd2:0xbe7:0x0 0x200000bd2:0xbec:0x0 0x200000bd2:0xbf3:0x0 0x200000bd2:0xc06:0x0 0x200000bd2:0xc0c:0x0 0x200000bd2:0xc12:0x0 0x200000bd2:0xc1c:0x0 0x200000bd2:0xc1f:0x0 0x200000bd2:0xc25:0x0 0x200000bd2:0xc28:0x0 0x200000bd2:0xc2e:0x0 0x200000bd2:0xc33:0x0 0x200000bd2:0xc37:0x0 0x200000bd2:0xc39:0x0 0x200000bd2:0xc93:0x0 0x200000bd2:0xc95:0x0 0x2000013a1:0x3:0x0 0x2000013a1:0x3ee:0x0 0x2000013a1:0x3f3:0x0 0x2000013a1:0x3f8:0x0 0x2000013a1:0x401:0x0 0x2000013a1:0x5:0x0 0x200001b71:0x1:0x0 500 /mnt/lustre/.lustre/trash/MDT0000/0: 0x200001b71:0x6:0x0 /mnt/lustre/.lustre/trash/MDT0000/0/0x200001b71:0x6:0x0: tfile.0 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd0:0x821:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbcc:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbd2:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe7:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbec:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbf3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc06:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc0c:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc12:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc1c:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc1f:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc25:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc28:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc2e:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc33:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc37:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc39:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc93:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc95:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x2000013a1:0x3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x2000013a1:0x3ee:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x2000013a1:0x3f3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x2000013a1:0x3f8:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x2000013a1:0x401:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x2000013a1:0x5:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200001b71:0x1:0x0: /mnt/lustre/.lustre/trash/MDT0000/500: /mnt/lustre/.lustre/trash/MDT0001: 0x240000404:0x799:0x0 0x240000404:0x79b:0x0 0x240000405:0x41:0x0 0x240000405:0xc:0x0 /mnt/lustre/.lustre/trash/MDT0001/0x240000404:0x799:0x0: /mnt/lustre/.lustre/trash/MDT0001/0x240000404:0x79b:0x0: /mnt/lustre/.lustre/trash/MDT0001/0x240000405:0x41:0x0: /mnt/lustre/.lustre/trash/MDT0001/0x240000405:0xc:0x0: On MGS 192.168.201.148, active = nodemap.active=1 waiting 10 secs for sync On MGS 192.168.201.148, tnm.trusted_nodemap = nodemap.tnm.trusted_nodemap=1 waiting 10 secs for sync nodemap.tnm.idmap= [ { idtype: uid, client_id: 500, fs_id: 10000 }, { idtype: uid, client_id: 501, fs_id: 10001 }, { idtype: uid, client_id: 502, fs_id: 10002 }, { idtype: uid, client_id: 503, fs_id: 10003 }, { idtype: uid, client_id: 504, fs_id: 10004 }, { idtype: uid, client_id: 505, fs_id: 10005 }, { idtype: uid, client_id: 506, fs_id: 10006 }, { idtype: uid, client_id: 507, fs_id: 10007 }, { idtype: uid, client_id: 508, fs_id: 10008 }, { idtype: uid, client_id: 509, fs_id: 10009 } ] /mnt/lustre/.lustre/trash: MDT0000 MDT0001 /mnt/lustre/.lustre/trash/MDT0000: 0 0x200000bd0:0x821:0x0 0x200000bd2:0xbcc:0x0 0x200000bd2:0xbd2:0x0 0x200000bd2:0xbe3:0x0 0x200000bd2:0xbe7:0x0 0x200000bd2:0xbec:0x0 0x200000bd2:0xbf3:0x0 0x200000bd2:0xc06:0x0 0x200000bd2:0xc0c:0x0 0x200000bd2:0xc12:0x0 0x200000bd2:0xc1c:0x0 0x200000bd2:0xc1f:0x0 0x200000bd2:0xc25:0x0 0x200000bd2:0xc28:0x0 0x200000bd2:0xc2e:0x0 0x200000bd2:0xc33:0x0 0x200000bd2:0xc37:0x0 0x200000bd2:0xc39:0x0 0x200000bd2:0xc93:0x0 0x200000bd2:0xc95:0x0 0x2000013a1:0x3:0x0 0x2000013a1:0x3ee:0x0 0x2000013a1:0x3f3:0x0 0x2000013a1:0x3f8:0x0 0x2000013a1:0x401:0x0 0x2000013a1:0x5:0x0 0x200001b71:0x1:0x0 500 tnm /mnt/lustre/.lustre/trash/MDT0000/0: 0x200001b71:0x6:0x0 /mnt/lustre/.lustre/trash/MDT0000/0/0x200001b71:0x6:0x0: tfile.0 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd0:0x821:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbcc:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbd2:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe7:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbec:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbf3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc06:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc0c:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc12:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc1c:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc1f:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc25:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc28:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc2e:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc33:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc37:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc39:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc93:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc95:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x2000013a1:0x3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x2000013a1:0x3ee:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x2000013a1:0x3f3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x2000013a1:0x3f8:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x2000013a1:0x401:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x2000013a1:0x5:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200001b71:0x1:0x0: /mnt/lustre/.lustre/trash/MDT0000/500: /mnt/lustre/.lustre/trash/MDT0000/tnm: 0 /mnt/lustre/.lustre/trash/MDT0000/tnm/0: 0x200001b71:0x6:0x0 /mnt/lustre/.lustre/trash/MDT0000/tnm/0/0x200001b71:0x6:0x0: tfile.1 /mnt/lustre/.lustre/trash/MDT0001: 0x240000404:0x799:0x0 0x240000404:0x79b:0x0 0x240000405:0x41:0x0 0x240000405:0xc:0x0 /mnt/lustre/.lustre/trash/MDT0001/0x240000404:0x799:0x0: /mnt/lustre/.lustre/trash/MDT0001/0x240000404:0x79b:0x0: /mnt/lustre/.lustre/trash/MDT0001/0x240000405:0x41:0x0: /mnt/lustre/.lustre/trash/MDT0001/0x240000405:0xc:0x0: running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d117za.sanityn/tfile.2] /mnt/lustre/.lustre/trash: MDT0000 MDT0001 /mnt/lustre/.lustre/trash/MDT0000: 0 0x200000bd0:0x821:0x0 0x200000bd2:0xbcc:0x0 0x200000bd2:0xbd2:0x0 0x200000bd2:0xbe3:0x0 0x200000bd2:0xbe7:0x0 0x200000bd2:0xbec:0x0 0x200000bd2:0xbf3:0x0 0x200000bd2:0xc06:0x0 0x200000bd2:0xc0c:0x0 0x200000bd2:0xc12:0x0 0x200000bd2:0xc1c:0x0 0x200000bd2:0xc1f:0x0 0x200000bd2:0xc25:0x0 0x200000bd2:0xc28:0x0 0x200000bd2:0xc2e:0x0 0x200000bd2:0xc33:0x0 0x200000bd2:0xc37:0x0 0x200000bd2:0xc39:0x0 0x200000bd2:0xc93:0x0 0x200000bd2:0xc95:0x0 0x2000013a1:0x3:0x0 0x2000013a1:0x3ee:0x0 0x2000013a1:0x3f3:0x0 0x2000013a1:0x3f8:0x0 0x2000013a1:0x401:0x0 0x2000013a1:0x5:0x0 0x200001b71:0x1:0x0 500 tnm /mnt/lustre/.lustre/trash/MDT0000/0: 0x200001b71:0x6:0x0 /mnt/lustre/.lustre/trash/MDT0000/0/0x200001b71:0x6:0x0: tfile.0 /mnt/lustre/.lustre/trash/MDT0000/0x200000bd0:0x821:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbcc:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbd2:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe7:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbec:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbf3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc06:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc0c:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc12:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc1c:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc1f:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc25:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc28:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc2e:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc33:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc37:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc39:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc93:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc95:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x2000013a1:0x3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x2000013a1:0x3ee:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x2000013a1:0x3f3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x2000013a1:0x3f8:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x2000013a1:0x401:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x2000013a1:0x5:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200001b71:0x1:0x0: /mnt/lustre/.lustre/trash/MDT0000/500: /mnt/lustre/.lustre/trash/MDT0000/tnm: 0 500 /mnt/lustre/.lustre/trash/MDT0000/tnm/0: 0x200001b71:0x6:0x0 /mnt/lustre/.lustre/trash/MDT0000/tnm/0/0x200001b71:0x6:0x0: tfile.1 /mnt/lustre/.lustre/trash/MDT0000/tnm/500: 0x200001b71:0x6:0x0 /mnt/lustre/.lustre/trash/MDT0000/tnm/500/0x200001b71:0x6:0x0: tfile.2 /mnt/lustre/.lustre/trash/MDT0001: 0x240000404:0x799:0x0 0x240000404:0x79b:0x0 0x240000405:0x41:0x0 0x240000405:0xc:0x0 /mnt/lustre/.lustre/trash/MDT0001/0x240000404:0x799:0x0: /mnt/lustre/.lustre/trash/MDT0001/0x240000404:0x79b:0x0: /mnt/lustre/.lustre/trash/MDT0001/0x240000405:0x41:0x0: /mnt/lustre/.lustre/trash/MDT0001/0x240000405:0xc:0x0: oleg148-server: debugfs 1.47.3-wc2 (11-Nov-2025) Inode: 520205 Type: regular Mode: 0644 Flags: 0x80002 Generation: 1391603252 Version: 0x00000004:000000fd User: -2 Group: -2 Project: -2 Size: 0 File ACL: 0 Links: 1 Blockcount: 0 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x694914f3:00000000 -- Mon Dec 22 04:52:51 2025 atime: 0x694914f3:00000000 -- Mon Dec 22 04:52:51 2025 mtime: 0x694914f3:00000000 -- Mon Dec 22 04:52:51 2025 crtime: 0x694914f4:686d6600 -- Mon Dec 22 04:52:52 2025 Size of extra inode fields: 32 Extended attributes: lma: fid=[0x200001b71:0x9:0x0] compat=0 incompat=0 trusted.lov (56) = d0 0b d1 0b 01 00 00 00 09 00 00 00 00 00 00 00 71 1b 00 00 02 00 00 00 00 00 40 00 01 00 00 00 01 04 00 c0 02 00 00 00 83 1c 00 00 00 00 00 00 00 00 00 00 01 00 00 00 user.job (9) = "touch.500" linkea: idx=0 parent=[0x200001b70:0x7:0x0] name='tfile.2' trusted.som (24) = 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 trusted.unrm (24) = 00 00 00 00 f4 01 00 00 f4 01 00 00 00 00 00 00 f4 14 49 69 00 00 00 00 EXTENTS: file: /mnt/lustre/.lustre/trash/MDT0000/tnm/500/0x200001b71:0x6:0x0/tfile.2, flags: 0X80002 unrm, uid: 500 gid: 500 projid: 0 timestamp: 1766397172 On MGS 192.168.201.148, active = nodemap.active=0 waiting 10 secs for sync mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117za (46s) == sanityn test 117zb: Test Trash Can for PLAIN Trash Can type with nodemap ========================================================== 04:53:08 (1766397188) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain /mnt/lustre/.lustre/trash: MDT0000 MDT0001 /mnt/lustre/.lustre/trash/MDT0000: 0 0x200000bd0:0x821:0x0 0x200000bd2:0xbcc:0x0 0x200000bd2:0xbd2:0x0 0x200000bd2:0xbe3:0x0 0x200000bd2:0xbe7:0x0 0x200000bd2:0xbec:0x0 0x200000bd2:0xbf3:0x0 0x200000bd2:0xc06:0x0 0x200000bd2:0xc0c:0x0 0x200000bd2:0xc12:0x0 0x200000bd2:0xc1c:0x0 0x200000bd2:0xc1f:0x0 0x200000bd2:0xc25:0x0 0x200000bd2:0xc28:0x0 0x200000bd2:0xc2e:0x0 0x200000bd2:0xc33:0x0 0x200000bd2:0xc37:0x0 0x200000bd2:0xc39:0x0 0x200000bd2:0xc93:0x0 0x200000bd2:0xc95:0x0 0x2000013a1:0x3:0x0 0x2000013a1:0x3ee:0x0 0x2000013a1:0x3f3:0x0 0x2000013a1:0x3f8:0x0 0x2000013a1:0x401:0x0 0x2000013a1:0x5:0x0 0x200001b71:0x1:0x0 0x200001b71:0xa:0x0 500 tnm /mnt/lustre/.lustre/trash/MDT0000/0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd0:0x821:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbcc:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbd2:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe7:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbec:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbf3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc06:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc0c:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc12:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc1c:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc1f:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc25:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc28:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc2e:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc33:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc37:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc39:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc93:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc95:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x2000013a1:0x3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x2000013a1:0x3ee:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x2000013a1:0x3f3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x2000013a1:0x3f8:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x2000013a1:0x401:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x2000013a1:0x5:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200001b71:0x1:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200001b71:0xa:0x0: tfile.0 /mnt/lustre/.lustre/trash/MDT0000/500: /mnt/lustre/.lustre/trash/MDT0000/tnm: /mnt/lustre/.lustre/trash/MDT0001: 0x240000404:0x799:0x0 0x240000404:0x79b:0x0 0x240000405:0x41:0x0 0x240000405:0xc:0x0 /mnt/lustre/.lustre/trash/MDT0001/0x240000404:0x799:0x0: /mnt/lustre/.lustre/trash/MDT0001/0x240000404:0x79b:0x0: /mnt/lustre/.lustre/trash/MDT0001/0x240000405:0x41:0x0: /mnt/lustre/.lustre/trash/MDT0001/0x240000405:0xc:0x0: On MGS 192.168.201.148, active = nodemap.active=1 waiting 10 secs for sync On MGS 192.168.201.148, tnm.trusted_nodemap = nodemap.tnm.trusted_nodemap=1 waiting 10 secs for sync nodemap.tnm.idmap= [ { idtype: uid, client_id: 500, fs_id: 10000 }, { idtype: uid, client_id: 501, fs_id: 10001 }, { idtype: uid, client_id: 502, fs_id: 10002 }, { idtype: uid, client_id: 503, fs_id: 10003 }, { idtype: uid, client_id: 504, fs_id: 10004 }, { idtype: uid, client_id: 505, fs_id: 10005 }, { idtype: uid, client_id: 506, fs_id: 10006 }, { idtype: uid, client_id: 507, fs_id: 10007 }, { idtype: uid, client_id: 508, fs_id: 10008 }, { idtype: uid, client_id: 509, fs_id: 10009 } ] /mnt/lustre/.lustre/trash: MDT0000 MDT0001 /mnt/lustre/.lustre/trash/MDT0000: 0 0x200000bd0:0x821:0x0 0x200000bd2:0xbcc:0x0 0x200000bd2:0xbd2:0x0 0x200000bd2:0xbe3:0x0 0x200000bd2:0xbe7:0x0 0x200000bd2:0xbec:0x0 0x200000bd2:0xbf3:0x0 0x200000bd2:0xc06:0x0 0x200000bd2:0xc0c:0x0 0x200000bd2:0xc12:0x0 0x200000bd2:0xc1c:0x0 0x200000bd2:0xc1f:0x0 0x200000bd2:0xc25:0x0 0x200000bd2:0xc28:0x0 0x200000bd2:0xc2e:0x0 0x200000bd2:0xc33:0x0 0x200000bd2:0xc37:0x0 0x200000bd2:0xc39:0x0 0x200000bd2:0xc93:0x0 0x200000bd2:0xc95:0x0 0x2000013a1:0x3:0x0 0x2000013a1:0x3ee:0x0 0x2000013a1:0x3f3:0x0 0x2000013a1:0x3f8:0x0 0x2000013a1:0x401:0x0 0x2000013a1:0x5:0x0 0x200001b71:0x1:0x0 0x200001b71:0xa:0x0 500 tnm /mnt/lustre/.lustre/trash/MDT0000/0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd0:0x821:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbcc:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbd2:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe7:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbec:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbf3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc06:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc0c:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc12:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc1c:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc1f:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc25:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc28:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc2e:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc33:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc37:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc39:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc93:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc95:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x2000013a1:0x3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x2000013a1:0x3ee:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x2000013a1:0x3f3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x2000013a1:0x3f8:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x2000013a1:0x401:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x2000013a1:0x5:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200001b71:0x1:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200001b71:0xa:0x0: tfile.0 /mnt/lustre/.lustre/trash/MDT0000/500: /mnt/lustre/.lustre/trash/MDT0000/tnm: 0x200001b71:0xa:0x0 /mnt/lustre/.lustre/trash/MDT0000/tnm/0x200001b71:0xa:0x0: tfile.1 /mnt/lustre/.lustre/trash/MDT0001: 0x240000404:0x799:0x0 0x240000404:0x79b:0x0 0x240000405:0x41:0x0 0x240000405:0xc:0x0 /mnt/lustre/.lustre/trash/MDT0001/0x240000404:0x799:0x0: /mnt/lustre/.lustre/trash/MDT0001/0x240000404:0x79b:0x0: /mnt/lustre/.lustre/trash/MDT0001/0x240000405:0x41:0x0: /mnt/lustre/.lustre/trash/MDT0001/0x240000405:0xc:0x0: running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d117zb.sanityn/tfile.2] /mnt/lustre/.lustre/trash: MDT0000 MDT0001 /mnt/lustre/.lustre/trash/MDT0000: 0 0x200000bd0:0x821:0x0 0x200000bd2:0xbcc:0x0 0x200000bd2:0xbd2:0x0 0x200000bd2:0xbe3:0x0 0x200000bd2:0xbe7:0x0 0x200000bd2:0xbec:0x0 0x200000bd2:0xbf3:0x0 0x200000bd2:0xc06:0x0 0x200000bd2:0xc0c:0x0 0x200000bd2:0xc12:0x0 0x200000bd2:0xc1c:0x0 0x200000bd2:0xc1f:0x0 0x200000bd2:0xc25:0x0 0x200000bd2:0xc28:0x0 0x200000bd2:0xc2e:0x0 0x200000bd2:0xc33:0x0 0x200000bd2:0xc37:0x0 0x200000bd2:0xc39:0x0 0x200000bd2:0xc93:0x0 0x200000bd2:0xc95:0x0 0x2000013a1:0x3:0x0 0x2000013a1:0x3ee:0x0 0x2000013a1:0x3f3:0x0 0x2000013a1:0x3f8:0x0 0x2000013a1:0x401:0x0 0x2000013a1:0x5:0x0 0x200001b71:0x1:0x0 0x200001b71:0xa:0x0 500 tnm /mnt/lustre/.lustre/trash/MDT0000/0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd0:0x821:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbcc:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbd2:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbe7:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbec:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xbf3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc06:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc0c:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc12:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc1c:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc1f:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc25:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc28:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc2e:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc33:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc37:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc39:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc93:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200000bd2:0xc95:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x2000013a1:0x3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x2000013a1:0x3ee:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x2000013a1:0x3f3:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x2000013a1:0x3f8:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x2000013a1:0x401:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x2000013a1:0x5:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200001b71:0x1:0x0: /mnt/lustre/.lustre/trash/MDT0000/0x200001b71:0xa:0x0: tfile.0 /mnt/lustre/.lustre/trash/MDT0000/500: /mnt/lustre/.lustre/trash/MDT0000/tnm: 0x200001b71:0xa:0x0 /mnt/lustre/.lustre/trash/MDT0000/tnm/0x200001b71:0xa:0x0: tfile.1 tfile.2 /mnt/lustre/.lustre/trash/MDT0001: 0x240000404:0x799:0x0 0x240000404:0x79b:0x0 0x240000405:0x41:0x0 0x240000405:0xc:0x0 /mnt/lustre/.lustre/trash/MDT0001/0x240000404:0x799:0x0: /mnt/lustre/.lustre/trash/MDT0001/0x240000404:0x79b:0x0: /mnt/lustre/.lustre/trash/MDT0001/0x240000405:0x41:0x0: /mnt/lustre/.lustre/trash/MDT0001/0x240000405:0xc:0x0: oleg148-server: debugfs 1.47.3-wc2 (11-Nov-2025) Inode: 520205 Type: regular Mode: 0644 Flags: 0x80002 Generation: 1831717794 Version: 0x00000004:0000021e User: -2 Group: -2 Project: -2 Size: 0 File ACL: 0 Links: 1 Blockcount: 0 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x69491520:00000000 -- Mon Dec 22 04:53:36 2025 atime: 0x69491520:00000000 -- Mon Dec 22 04:53:36 2025 mtime: 0x69491520:00000000 -- Mon Dec 22 04:53:36 2025 crtime: 0x69491520:cf6c9600 -- Mon Dec 22 04:53:36 2025 Size of extra inode fields: 32 Extended attributes: lma: fid=[0x200001b71:0xd:0x0] compat=0 incompat=0 trusted.lov (56) = d0 0b d1 0b 01 00 00 00 0d 00 00 00 00 00 00 00 71 1b 00 00 02 00 00 00 00 00 40 00 01 00 00 00 01 04 00 80 02 00 00 00 c5 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 user.job (9) = "touch.500" linkea: idx=0 parent=[0x200001b70:0x9:0x0] name='tfile.2' trusted.som (24) = 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 trusted.unrm (24) = 00 00 00 00 f4 01 00 00 f4 01 00 00 00 00 00 00 20 15 49 69 00 00 00 00 EXTENTS: file: /mnt/lustre/.lustre/trash/MDT0000/tnm//0x200001b71:0xa:0x0/tfile.2, flags: 0X80002 unrm, uid: 500 gid: 500 projid: 0 timestamp: 1766397216 On MGS 192.168.201.148, active = nodemap.active=0 waiting 10 secs for sync mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117zb (44s) == sanityn test 117zc: TCU: Test striped dir for UID Trash Can type ========================================================== 04:53:52 (1766397232) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid Trash Can Type: uid uid Creating 100 files under striped directory: /mnt/lustre/d117zc.sanityn/striped_dir total: 100 open/close in 0.56 seconds: 177.02 ops/second f117zc.sanityn0 -> 1 f117zc.sanityn1 -> 1 f117zc.sanityn2 -> 1 f117zc.sanityn3 -> 1 f117zc.sanityn4 -> 1 f117zc.sanityn5 -> 1 f117zc.sanityn6 -> 1 f117zc.sanityn7 -> 1 f117zc.sanityn8 -> 1 f117zc.sanityn9 -> 1 f117zc.sanityn10 -> 0 f117zc.sanityn11 -> 0 f117zc.sanityn12 -> 0 f117zc.sanityn13 -> 1 f117zc.sanityn14 -> 0 f117zc.sanityn15 -> 1 f117zc.sanityn16 -> 1 f117zc.sanityn17 -> 0 f117zc.sanityn18 -> 1 f117zc.sanityn19 -> 1 f117zc.sanityn20 -> 0 f117zc.sanityn21 -> 1 f117zc.sanityn22 -> 0 f117zc.sanityn23 -> 0 f117zc.sanityn24 -> 0 f117zc.sanityn25 -> 1 f117zc.sanityn26 -> 0 f117zc.sanityn27 -> 0 f117zc.sanityn28 -> 1 f117zc.sanityn29 -> 1 f117zc.sanityn30 -> 1 f117zc.sanityn31 -> 0 f117zc.sanityn32 -> 0 f117zc.sanityn33 -> 0 f117zc.sanityn34 -> 0 f117zc.sanityn35 -> 1 f117zc.sanityn36 -> 0 f117zc.sanityn37 -> 1 f117zc.sanityn38 -> 1 f117zc.sanityn39 -> 1 f117zc.sanityn40 -> 1 f117zc.sanityn41 -> 1 f117zc.sanityn42 -> 0 f117zc.sanityn43 -> 1 f117zc.sanityn44 -> 0 f117zc.sanityn45 -> 0 f117zc.sanityn46 -> 0 f117zc.sanityn47 -> 0 f117zc.sanityn48 -> 0 f117zc.sanityn49 -> 1 f117zc.sanityn50 -> 0 f117zc.sanityn51 -> 0 f117zc.sanityn52 -> 0 f117zc.sanityn53 -> 1 f117zc.sanityn54 -> 0 f117zc.sanityn55 -> 1 f117zc.sanityn56 -> 0 f117zc.sanityn57 -> 1 f117zc.sanityn58 -> 1 f117zc.sanityn59 -> 0 f117zc.sanityn60 -> 1 f117zc.sanityn61 -> 0 f117zc.sanityn62 -> 0 f117zc.sanityn63 -> 0 f117zc.sanityn64 -> 1 f117zc.sanityn65 -> 0 f117zc.sanityn66 -> 0 f117zc.sanityn67 -> 1 f117zc.sanityn68 -> 1 f117zc.sanityn69 -> 1 f117zc.sanityn70 -> 0 f117zc.sanityn71 -> 0 f117zc.sanityn72 -> 0 f117zc.sanityn73 -> 0 f117zc.sanityn74 -> 1 f117zc.sanityn75 -> 0 f117zc.sanityn76 -> 1 f117zc.sanityn77 -> 0 f117zc.sanityn78 -> 0 f117zc.sanityn79 -> 1 f117zc.sanityn80 -> 0 f117zc.sanityn81 -> 1 f117zc.sanityn82 -> 1 f117zc.sanityn83 -> 1 f117zc.sanityn84 -> 0 f117zc.sanityn85 -> 1 f117zc.sanityn86 -> 1 f117zc.sanityn87 -> 1 f117zc.sanityn88 -> 0 f117zc.sanityn89 -> 1 f117zc.sanityn90 -> 0 f117zc.sanityn91 -> 1 f117zc.sanityn92 -> 1 f117zc.sanityn93 -> 0 f117zc.sanityn94 -> 0 f117zc.sanityn95 -> 0 f117zc.sanityn96 -> 1 f117zc.sanityn97 -> 0 f117zc.sanityn98 -> 0 f117zc.sanityn99 -> 1 #0 -> 0x200001b70:0xa:0x0 #1 -> 0x2400013a0:0x1:0x0 - unlinked 0 (time 1766397237 ; total 0 ; last 0) total: 100 unlinks in 0 seconds: inf unlinks/second f117zc.sanityn0 f117zc.sanityn1 f117zc.sanityn10 f117zc.sanityn11 f117zc.sanityn12 f117zc.sanityn13 f117zc.sanityn14 f117zc.sanityn15 f117zc.sanityn16 f117zc.sanityn17 f117zc.sanityn18 f117zc.sanityn19 f117zc.sanityn2 f117zc.sanityn20 f117zc.sanityn21 f117zc.sanityn22 f117zc.sanityn23 f117zc.sanityn24 f117zc.sanityn25 f117zc.sanityn26 f117zc.sanityn27 f117zc.sanityn28 f117zc.sanityn29 f117zc.sanityn3 f117zc.sanityn30 f117zc.sanityn31 f117zc.sanityn32 f117zc.sanityn33 f117zc.sanityn34 f117zc.sanityn35 f117zc.sanityn36 f117zc.sanityn37 f117zc.sanityn38 f117zc.sanityn39 f117zc.sanityn4 f117zc.sanityn40 f117zc.sanityn41 f117zc.sanityn42 f117zc.sanityn43 f117zc.sanityn44 f117zc.sanityn45 f117zc.sanityn46 f117zc.sanityn47 f117zc.sanityn48 f117zc.sanityn49 f117zc.sanityn5 f117zc.sanityn50 f117zc.sanityn51 f117zc.sanityn52 f117zc.sanityn53 f117zc.sanityn54 f117zc.sanityn55 f117zc.sanityn56 f117zc.sanityn57 f117zc.sanityn58 f117zc.sanityn59 f117zc.sanityn6 f117zc.sanityn60 f117zc.sanityn61 f117zc.sanityn62 f117zc.sanityn63 f117zc.sanityn64 f117zc.sanityn65 f117zc.sanityn66 f117zc.sanityn67 f117zc.sanityn68 f117zc.sanityn69 f117zc.sanityn7 f117zc.sanityn70 f117zc.sanityn71 f117zc.sanityn72 f117zc.sanityn73 f117zc.sanityn74 f117zc.sanityn75 f117zc.sanityn76 f117zc.sanityn77 f117zc.sanityn78 f117zc.sanityn79 f117zc.sanityn8 f117zc.sanityn80 f117zc.sanityn81 f117zc.sanityn82 f117zc.sanityn83 f117zc.sanityn84 f117zc.sanityn85 f117zc.sanityn86 f117zc.sanityn87 f117zc.sanityn88 f117zc.sanityn89 f117zc.sanityn9 f117zc.sanityn90 f117zc.sanityn91 f117zc.sanityn92 f117zc.sanityn93 f117zc.sanityn94 f117zc.sanityn95 f117zc.sanityn96 f117zc.sanityn97 f117zc.sanityn98 f117zc.sanityn99 f117zc.sanityn10 f117zc.sanityn11 f117zc.sanityn12 f117zc.sanityn13 f117zc.sanityn14 f117zc.sanityn15 f117zc.sanityn16 f117zc.sanityn17 f117zc.sanityn18 f117zc.sanityn19 f117zc.sanityn20 f117zc.sanityn21 f117zc.sanityn22 f117zc.sanityn23 f117zc.sanityn24 f117zc.sanityn25 f117zc.sanityn26 f117zc.sanityn27 f117zc.sanityn28 f117zc.sanityn29 f117zc.sanityn30 f117zc.sanityn31 f117zc.sanityn32 f117zc.sanityn33 f117zc.sanityn34 f117zc.sanityn35 f117zc.sanityn36 f117zc.sanityn37 f117zc.sanityn38 f117zc.sanityn39 f117zc.sanityn40 f117zc.sanityn41 f117zc.sanityn42 f117zc.sanityn43 f117zc.sanityn44 f117zc.sanityn45 f117zc.sanityn46 f117zc.sanityn47 f117zc.sanityn48 f117zc.sanityn49 f117zc.sanityn50 f117zc.sanityn51 f117zc.sanityn52 f117zc.sanityn53 f117zc.sanityn54 f117zc.sanityn55 f117zc.sanityn56 f117zc.sanityn57 f117zc.sanityn58 f117zc.sanityn59 f117zc.sanityn60 f117zc.sanityn61 f117zc.sanityn62 f117zc.sanityn63 f117zc.sanityn64 f117zc.sanityn65 f117zc.sanityn66 f117zc.sanityn67 f117zc.sanityn68 f117zc.sanityn69 f117zc.sanityn70 f117zc.sanityn71 f117zc.sanityn72 f117zc.sanityn73 f117zc.sanityn74 f117zc.sanityn75 f117zc.sanityn76 f117zc.sanityn77 f117zc.sanityn78 f117zc.sanityn79 f117zc.sanityn80 f117zc.sanityn81 f117zc.sanityn82 f117zc.sanityn83 f117zc.sanityn84 f117zc.sanityn85 f117zc.sanityn86 f117zc.sanityn87 f117zc.sanityn88 f117zc.sanityn89 f117zc.sanityn90 f117zc.sanityn91 f117zc.sanityn92 f117zc.sanityn93 f117zc.sanityn94 f117zc.sanityn95 f117zc.sanityn96 f117zc.sanityn97 f117zc.sanityn98 f117zc.sanityn99 mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117zc (13s) == sanityn test 117zd: TCU: Test striped dir for UID Trash Can type with nodemap ========================================================== 04:54:05 (1766397245) On MGS 192.168.201.148, active = nodemap.active=1 waiting 10 secs for sync On MGS 192.168.201.148, tnm.trusted_nodemap = nodemap.tnm.trusted_nodemap=1 waiting 10 secs for sync nodemap.tnm.idmap= [ { idtype: uid, client_id: 500, fs_id: 10000 }, { idtype: uid, client_id: 501, fs_id: 10001 }, { idtype: uid, client_id: 502, fs_id: 10002 }, { idtype: uid, client_id: 503, fs_id: 10003 }, { idtype: uid, client_id: 504, fs_id: 10004 }, { idtype: uid, client_id: 505, fs_id: 10005 }, { idtype: uid, client_id: 506, fs_id: 10006 }, { idtype: uid, client_id: 507, fs_id: 10007 }, { idtype: uid, client_id: 508, fs_id: 10008 }, { idtype: uid, client_id: 509, fs_id: 10009 } ] mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid Trash Can Type: uid uid Creating 100 files under striped directory: /mnt/lustre/d117zd.sanityn/striped_dir total: 100 open/close in 0.47 seconds: 214.75 ops/second f117zd.sanityn0 -> 0 f117zd.sanityn1 -> 0 f117zd.sanityn2 -> 0 f117zd.sanityn3 -> 0 f117zd.sanityn4 -> 0 f117zd.sanityn5 -> 0 f117zd.sanityn6 -> 0 f117zd.sanityn7 -> 0 f117zd.sanityn8 -> 0 f117zd.sanityn9 -> 0 f117zd.sanityn10 -> 0 f117zd.sanityn11 -> 1 f117zd.sanityn12 -> 1 f117zd.sanityn13 -> 0 f117zd.sanityn14 -> 1 f117zd.sanityn15 -> 1 f117zd.sanityn16 -> 1 f117zd.sanityn17 -> 1 f117zd.sanityn18 -> 1 f117zd.sanityn19 -> 0 f117zd.sanityn20 -> 1 f117zd.sanityn21 -> 0 f117zd.sanityn22 -> 0 f117zd.sanityn23 -> 1 f117zd.sanityn24 -> 1 f117zd.sanityn25 -> 0 f117zd.sanityn26 -> 1 f117zd.sanityn27 -> 1 f117zd.sanityn28 -> 1 f117zd.sanityn29 -> 0 f117zd.sanityn30 -> 1 f117zd.sanityn31 -> 0 f117zd.sanityn32 -> 1 f117zd.sanityn33 -> 0 f117zd.sanityn34 -> 1 f117zd.sanityn35 -> 0 f117zd.sanityn36 -> 0 f117zd.sanityn37 -> 0 f117zd.sanityn38 -> 0 f117zd.sanityn39 -> 1 f117zd.sanityn40 -> 1 f117zd.sanityn41 -> 0 f117zd.sanityn42 -> 1 f117zd.sanityn43 -> 1 f117zd.sanityn44 -> 0 f117zd.sanityn45 -> 0 f117zd.sanityn46 -> 1 f117zd.sanityn47 -> 0 f117zd.sanityn48 -> 1 f117zd.sanityn49 -> 0 f117zd.sanityn50 -> 1 f117zd.sanityn51 -> 1 f117zd.sanityn52 -> 1 f117zd.sanityn53 -> 0 f117zd.sanityn54 -> 0 f117zd.sanityn55 -> 0 f117zd.sanityn56 -> 0 f117zd.sanityn57 -> 1 f117zd.sanityn58 -> 1 f117zd.sanityn59 -> 1 f117zd.sanityn60 -> 1 f117zd.sanityn61 -> 0 f117zd.sanityn62 -> 0 f117zd.sanityn63 -> 0 f117zd.sanityn64 -> 0 f117zd.sanityn65 -> 1 f117zd.sanityn66 -> 0 f117zd.sanityn67 -> 0 f117zd.sanityn68 -> 1 f117zd.sanityn69 -> 1 f117zd.sanityn70 -> 1 f117zd.sanityn71 -> 1 f117zd.sanityn72 -> 0 f117zd.sanityn73 -> 0 f117zd.sanityn74 -> 0 f117zd.sanityn75 -> 0 f117zd.sanityn76 -> 1 f117zd.sanityn77 -> 1 f117zd.sanityn78 -> 1 f117zd.sanityn79 -> 0 f117zd.sanityn80 -> 0 f117zd.sanityn81 -> 0 f117zd.sanityn82 -> 1 f117zd.sanityn83 -> 1 f117zd.sanityn84 -> 1 f117zd.sanityn85 -> 0 f117zd.sanityn86 -> 0 f117zd.sanityn87 -> 1 f117zd.sanityn88 -> 0 f117zd.sanityn89 -> 0 f117zd.sanityn90 -> 0 f117zd.sanityn91 -> 0 f117zd.sanityn92 -> 1 f117zd.sanityn93 -> 1 f117zd.sanityn94 -> 0 f117zd.sanityn95 -> 1 f117zd.sanityn96 -> 0 f117zd.sanityn97 -> 1 f117zd.sanityn98 -> 1 f117zd.sanityn99 -> 1 #0 -> 0x200001b70:0xe:0x0 #1 -> 0x2400013a0:0x4:0x0 - unlinked 0 (time 1766397276 ; total 0 ; last 0) total: 100 unlinks in 1 seconds: 100.000000 unlinks/second f117zd.sanityn0 f117zd.sanityn1 f117zd.sanityn10 f117zd.sanityn11 f117zd.sanityn12 f117zd.sanityn13 f117zd.sanityn14 f117zd.sanityn15 f117zd.sanityn16 f117zd.sanityn17 f117zd.sanityn18 f117zd.sanityn19 f117zd.sanityn2 f117zd.sanityn20 f117zd.sanityn21 f117zd.sanityn22 f117zd.sanityn23 f117zd.sanityn24 f117zd.sanityn25 f117zd.sanityn26 f117zd.sanityn27 f117zd.sanityn28 f117zd.sanityn29 f117zd.sanityn3 f117zd.sanityn30 f117zd.sanityn31 f117zd.sanityn32 f117zd.sanityn33 f117zd.sanityn34 f117zd.sanityn35 f117zd.sanityn36 f117zd.sanityn37 f117zd.sanityn38 f117zd.sanityn39 f117zd.sanityn4 f117zd.sanityn40 f117zd.sanityn41 f117zd.sanityn42 f117zd.sanityn43 f117zd.sanityn44 f117zd.sanityn45 f117zd.sanityn46 f117zd.sanityn47 f117zd.sanityn48 f117zd.sanityn49 f117zd.sanityn5 f117zd.sanityn50 f117zd.sanityn51 f117zd.sanityn52 f117zd.sanityn53 f117zd.sanityn54 f117zd.sanityn55 f117zd.sanityn56 f117zd.sanityn57 f117zd.sanityn58 f117zd.sanityn59 f117zd.sanityn6 f117zd.sanityn60 f117zd.sanityn61 f117zd.sanityn62 f117zd.sanityn63 f117zd.sanityn64 f117zd.sanityn65 f117zd.sanityn66 f117zd.sanityn67 f117zd.sanityn68 f117zd.sanityn69 f117zd.sanityn7 f117zd.sanityn70 f117zd.sanityn71 f117zd.sanityn72 f117zd.sanityn73 f117zd.sanityn74 f117zd.sanityn75 f117zd.sanityn76 f117zd.sanityn77 f117zd.sanityn78 f117zd.sanityn79 f117zd.sanityn8 f117zd.sanityn80 f117zd.sanityn81 f117zd.sanityn82 f117zd.sanityn83 f117zd.sanityn84 f117zd.sanityn85 f117zd.sanityn86 f117zd.sanityn87 f117zd.sanityn88 f117zd.sanityn89 f117zd.sanityn9 f117zd.sanityn90 f117zd.sanityn91 f117zd.sanityn92 f117zd.sanityn93 f117zd.sanityn94 f117zd.sanityn95 f117zd.sanityn96 f117zd.sanityn97 f117zd.sanityn98 f117zd.sanityn99 f117zd.sanityn10 f117zd.sanityn11 f117zd.sanityn12 f117zd.sanityn13 f117zd.sanityn14 f117zd.sanityn15 f117zd.sanityn16 f117zd.sanityn17 f117zd.sanityn18 f117zd.sanityn19 f117zd.sanityn20 f117zd.sanityn21 f117zd.sanityn22 f117zd.sanityn23 f117zd.sanityn24 f117zd.sanityn25 f117zd.sanityn26 f117zd.sanityn27 f117zd.sanityn28 f117zd.sanityn29 f117zd.sanityn30 f117zd.sanityn31 f117zd.sanityn32 f117zd.sanityn33 f117zd.sanityn34 f117zd.sanityn35 f117zd.sanityn36 f117zd.sanityn37 f117zd.sanityn38 f117zd.sanityn39 f117zd.sanityn40 f117zd.sanityn41 f117zd.sanityn42 f117zd.sanityn43 f117zd.sanityn44 f117zd.sanityn45 f117zd.sanityn46 f117zd.sanityn47 f117zd.sanityn48 f117zd.sanityn49 f117zd.sanityn50 f117zd.sanityn51 f117zd.sanityn52 f117zd.sanityn53 f117zd.sanityn54 f117zd.sanityn55 f117zd.sanityn56 f117zd.sanityn57 f117zd.sanityn58 f117zd.sanityn59 f117zd.sanityn60 f117zd.sanityn61 f117zd.sanityn62 f117zd.sanityn63 f117zd.sanityn64 f117zd.sanityn65 f117zd.sanityn66 f117zd.sanityn67 f117zd.sanityn68 f117zd.sanityn69 f117zd.sanityn70 f117zd.sanityn71 f117zd.sanityn72 f117zd.sanityn73 f117zd.sanityn74 f117zd.sanityn75 f117zd.sanityn76 f117zd.sanityn77 f117zd.sanityn78 f117zd.sanityn79 f117zd.sanityn80 f117zd.sanityn81 f117zd.sanityn82 f117zd.sanityn83 f117zd.sanityn84 f117zd.sanityn85 f117zd.sanityn86 f117zd.sanityn87 f117zd.sanityn88 f117zd.sanityn89 f117zd.sanityn90 f117zd.sanityn91 f117zd.sanityn92 f117zd.sanityn93 f117zd.sanityn94 f117zd.sanityn95 f117zd.sanityn96 f117zd.sanityn97 f117zd.sanityn98 f117zd.sanityn99 mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid On MGS 192.168.201.148, active = nodemap.active=0 waiting 10 secs for sync PASS 117zd (51s) == sanityn test 118Aa: TCU: Orphan object handling for a regular file or an empty dir ========================================================== 04:54:56 (1766397296) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain oleg148-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg148-server: debugfs 1.47.3-wc2 (11-Nov-2025) 520208 (4096) 0x200001b71:0x78:0x0 Waiting for MDT destroys to complete oleg148-server: debugfs 1.47.3-wc2 (11-Nov-2025) 0 (4096) 0x200001b71:0x78:0x0 sanityn test_118Aa: @@@@@@ FAIL: 0x200001b71:0x78:0x0 should be removed from PENDING Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7318:error() = /home/green/git/lustre-release/lustre/tests/sanityn.sh:9184:test_118Aa() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7694:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7757:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7572:run_test() = /home/green/git/lustre-release/lustre/tests/sanityn.sh:9206:main() Dumping lctl log to /tmp/testlogs//sanityn.test_118Aa.*.1766397314.log rsync: chown "/tmp/testlogs/.sanityn.test_118Aa.debug_log.oleg148-server.1766397314.log.TqBrKr" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanityn.test_118Aa.dmesg.oleg148-server.1766397314.log.wDxOuR" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid FAIL 118Aa (21s) == sanityn test 118Ab: TCU: Orphan object handling for non-emtpy dir ========================================================== 04:55:17 (1766397317) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain oleg148-server: debugfs 1.47.3-wc2 (11-Nov-2025) 64004 (12) . 2 (4084) .. 520210 (4096) 0x200001b71:0x7a:0x0 Waiting for MDT destroys to complete oleg148-server: debugfs 1.47.3-wc2 (11-Nov-2025) 64004 (12) . 2 (4084) .. 0 (4096) 0x200001b71:0x7a:0x0 sanityn test_118Ab: @@@@@@ FAIL: 0x200001b71:0x7a:0x0 should be removed/cleanup from PENDING Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7318:error() = /home/green/git/lustre-release/lustre/tests/sanityn.sh:9250:test_118Ab() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7694:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7757:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7572:run_test() = /home/green/git/lustre-release/lustre/tests/sanityn.sh:9254:main() Dumping lctl log to /tmp/testlogs//sanityn.test_118Ab.*.1766397335.log rsync: chown "/tmp/testlogs/.sanityn.test_118Ab.debug_log.oleg148-server.1766397335.log.LNfE6t" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanityn.test_118Ab.dmesg.oleg148-server.1766397335.log.LOuR7b" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid FAIL 118Ab (20s) == sanityn test 118Ba: TCU: 'oldest' repeated deletion for a regular file or empty dir ========================================================== 04:55:37 (1766397337) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain mdd.lustre-MDT0000.trash_can_delete_policy=oldest mdd.lustre-MDT0001.trash_can_delete_policy=oldest llite.lustre-ffff9e1b87e1e800.trash_file_read=1 llite.lustre-ffff9e1b8ac5a800.trash_file_read=1 llite.lustre-ffff9e1b87e1e800.trash_file_read=0 llite.lustre-ffff9e1b8ac5a800.trash_file_read=0 mdd.lustre-MDT0000.trash_can_delete_policy=timestamp mdd.lustre-MDT0001.trash_can_delete_policy=timestamp mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 118Ba (8s) == sanityn test 118Bb: TCU: 'timestamp' repeated deletion policy ========================================================== 04:55:45 (1766397345) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain mdd.lustre-MDT0000.trash_can_delete_policy=timestamp mdd.lustre-MDT0001.trash_can_delete_policy=timestamp f118Bb.sanityn f118Bb.sanityn.2025-12-22T09:55:49 f118Bb.sanityn.2025-12-22T09:55:49.490352471 f118Bb.sanityn.2025-12-22T09:55:49.515110500 f118Bb.sanityn.2025-12-22T09:55:49.544783852 f118Bb.sanityn.2025-12-22T09:55:49.571067823 f118Bb.sanityn.2025-12-22T09:55:49.595745834 f118Bb.sanityn.2025-12-22T09:55:49.619504324 f118Bb.sanityn.2025-12-22T09:55:49.649797433 f118Bb.sanityn.2025-12-22T09:55:49.681959880 f118Bb.sanityn.2025-12-22T09:55:49.719690693 f118Bb.sanityn.2025-12-22T09:55:49.746050630 f118Bb.sanityn.2025-12-22T09:55:49.768320326 f118Bb.sanityn.2025-12-22T09:55:49.790454786 f118Bb.sanityn.2025-12-22T09:55:49.816302360 f118Bb.sanityn.2025-12-22T09:55:49.841878401 d118Bb.sanityn d118Bb.sanityn.2025-12-22T09:55:50 d118Bb.sanityn.2025-12-22T09:55:50.646615585 d118Bb.sanityn.2025-12-22T09:55:50.668968536 d118Bb.sanityn.2025-12-22T09:55:50.691908418 d118Bb.sanityn.2025-12-22T09:55:50.713072013 d118Bb.sanityn.2025-12-22T09:55:50.737835983 d118Bb.sanityn.2025-12-22T09:55:50.758674463 d118Bb.sanityn.2025-12-22T09:55:50.778560571 d118Bb.sanityn.2025-12-22T09:55:50.797718412 d118Bb.sanityn.2025-12-22T09:55:50.815427593 d118Bb.sanityn.2025-12-22T09:55:50.834177751 d118Bb.sanityn.2025-12-22T09:55:50.854441402 d118Bb.sanityn.2025-12-22T09:55:50.875462917 d118Bb.sanityn.2025-12-22T09:55:50.896935753 d118Bb.sanityn.2025-12-22T09:55:50.917414143 d118Bb.sanityn.mulevel d118Bb.sanityn.mulevel.2025-12-22T09:55:51 d118Bb.sanityn.mulevel.2025-12-22T09:55:51.803812670 d118Bb.sanityn.mulevel.2025-12-22T09:55:51.908878936 d118Bb.sanityn.mulevel.2025-12-22T09:55:52 d118Bb.sanityn.mulevel.2025-12-22T09:55:52.125682057 d118Bb.sanityn.mulevel.2025-12-22T09:55:52.262629399 d118Bb.sanityn.mulevel.2025-12-22T09:55:52.388378524 d118Bb.sanityn.mulevel.2025-12-22T09:55:52.512311453 d118Bb.sanityn.mulevel.2025-12-22T09:55:52.634185628 d118Bb.sanityn.mulevel.2025-12-22T09:55:52.754864714 d118Bb.sanityn.mulevel.2025-12-22T09:55:52.883300645 d118Bb.sanityn.mulevel.2025-12-22T09:55:52.993365926 d118Bb.sanityn.mulevel.2025-12-22T09:55:53 d118Bb.sanityn.mulevel.2025-12-22T09:55:53.254182977 d118Bb.sanityn.mulevel.2025-12-22T09:55:53.413300904 mdd.lustre-MDT0000.trash_can_delete_policy=timestamp mdd.lustre-MDT0001.trash_can_delete_policy=timestamp mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 118Bb (13s) == sanityn test 118C: TCU: add support for rename() ====== 04:55:58 (1766397358) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain mdd.lustre-MDT0000.trash_can_delete_policy=timestamp mdd.lustre-MDT0001.trash_can_delete_policy=timestamp f118C.sanityn f118C.sanityn f118C.sanityn f118C.sanityn.2025-12-22T09:56:02 f118C.sanityn.2025-12-22T09:56:02.268382686 mdd.lustre-MDT0000.trash_can_delete_policy=timestamp mdd.lustre-MDT0001.trash_can_delete_policy=timestamp mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 118C (7s) == sanityn test 118d: TCU: repeated deletion of same fname for a empty dir in DNE env ========================================================== 04:56:05 (1766397365) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain mdd.lustre-MDT0000.trash_can_delete_policy=timestamp mdd.lustre-MDT0001.trash_can_delete_policy=timestamp d118d.sanityn.0 d118d.sanityn.0.2025-12-22T09:56:09 d118d.sanityn.0.2025-12-22T09:56:09.820081486 d118d.sanityn.0.2025-12-22T09:56:09.850060064 d118d.sanityn.0.2025-12-22T09:56:09.883740723 d118d.sanityn.0.2025-12-22T09:56:09.915587044 d118d.sanityn.0.2025-12-22T09:56:09.949508930 d118d.sanityn.0.2025-12-22T09:56:09.988242092 d118d.sanityn.0.2025-12-22T09:56:10 d118d.sanityn.0.2025-12-22T09:56:10.050536739 mdd.lustre-MDT0000.trash_can_delete_policy=timestamp mdd.lustre-MDT0001.trash_can_delete_policy=timestamp mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 118d (8s) == sanityn test 118e: TCU: Repeated deletion of same name for a non-empty remote dir ========================================================== 04:56:13 (1766397373) SKIP: sanityn test_118e needs 3 MDTs at least SKIP 118e (1s) == sanityn test 118f: Avoid caching negative dentry cache for dir in Trash Can ========================================================== 04:56:15 (1766397375) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain plain /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 9492: tree: command not found stat: cannot statx '/mnt/lustre/.lustre/trash/MDT0000/0x200001b71:0xf0:0x0/f118f.sanityn.1': No such file or directory === 2nd rm /mnt/lustre/d118f.sanityn/f118f.sanityn.1 ==== total 8 d--------- 2 root root 4096 Dec 22 2025 . d--------- 32 root root 4096 Dec 22 02:28 .. -rw-r--r-- 1 root root 0 Dec 22 04:56 f118f.sanityn.1 mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 118f (7s) == sanityn test 200: service remains healthy while able to process request ========================================================== 04:56:21 (1766397381) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=4 striped dir -i0 -c2 -H crush2 /mnt/lustre/d200.sanityn fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0866 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0862 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 52.2645 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 52.2677 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 84.0203 s, 0.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 84.0315 s, 0.0 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 116.799 s, 0.0 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 116.796 s, 0.0 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg148-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9e1b87e1e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e1b87e1e800.ost_server_uuid in IDLE state after 0 sec PASS 200 (124s) cleanup: ====================================================== == sanityn test complete, duration 8947 sec ============== 04:58:26 (1766397506) sanityn: FAIL: test_117yb /etc and /mnt/lustre/d117yb.sanityn/etc diff sanityn: FAIL: test_118Aa 0x200001b71:0x78:0x0 should be removed from PENDING sanityn: FAIL: test_118Ab 0x200001b71:0x7a:0x0 should be removed/cleanup from PENDING === sanityn: start cleanup 04:58:26 (1766397506) === Stopping clients: oleg148-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg148-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 05:00:34 (1766397634) ===