-----============= acceptance-small: sanityn ============----- Sun May 17 21:26:00 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg135-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 21:26:29 (1779067589) === Starting client oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 Mount client oleg135-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 Started clients oleg135-client.virtnet: 192.168.201.135@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg135-client.virtnet: executing check_config_client /mnt/lustre oleg135-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg135-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffffa0e708efa800.idle_timeout=debug osc.lustre-OST0000-osc-ffffa0e71017f800.idle_timeout=debug osc.lustre-OST0001-osc-ffffa0e708efa800.idle_timeout=debug osc.lustre-OST0001-osc-ffffa0e71017f800.idle_timeout=debug disable quota as required oleg135-server: oleg135-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanityn: finish setup 21:27:15 (1779067635) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577543 s, 18.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8189] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 21:27:18 (1779067638) PASS 0 (10s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 21:27:28 (1779067648) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (9s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 21:27:37 (1779067657) -rw-r--r-- 1 root root 0 May 17 21:27 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (7s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 21:27:45 (1779067665) -rw-r--r-- 1 root root 0 May 17 21:27 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (7s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 21:27:52 (1779067672) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (7s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 21:27:59 (1779067679) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (8s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 21:28:07 (1779067687) total 0 -rwxrwxrwx 1 root root 0 May 17 21:27 f2a -rwxrwxrwx 1 root root 0 May 17 21:27 f2b total 0 -rwxrwxrwx 1 root root 0 May 17 21:27 f2a -rwxrwxrwx 1 root root 0 May 17 21:27 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0115017 s, 44.5 kB/s PASS 2e (9s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 21:28:17 (1779067697) /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn has type file OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn has perms 0777 OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn is owned by user #500 OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn is owned by group #500 OK Can't lstat /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn: No such file or directory PASS 2f (8s) == sanityn test 2g: check blocks update on sync write ==== 21:28:25 (1779067705) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.20095 s, 10.4 MB/s /mnt/lustre/f2g.sanityn has 4096 blocks /mnt/lustre2/f2g.sanityn has 4096 blocks PASS 2g (8s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 21:28:33 (1779067713) PASS 3 (9s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 21:28:42 (1779067722) PASS 4 (9s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 21:28:51 (1779067731) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (7s) == sanityn test 6: remove of open file on other node ============================================================================ 21:28:59 (1779067739) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (6s) == sanityn test 7: remove of open directory on other node ======================================================================= 21:29:05 (1779067745) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (8s) == sanityn test 8: remove of open special file on other node ==================================================================== 21:29:13 (1779067753) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (7s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 21:29:20 (1779067760) PASS 9a (8s) == sanityn test 9b: append to striped sparse file ======== 21:29:28 (1779067768) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358218 s, 29.3 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0625173 s, 0.0 kB/s Data read (expecting 'foo'): foo PASS 9b (8s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 21:29:36 (1779067776) 1+0 records in 1+0 records out 1 byte copied, 0.0140165 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0278697 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0161306 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.010534 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0179403 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0273378 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0454402 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0204735 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0214707 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0240274 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0222322 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0321366 s, 0.0 kB/s PASS 10a (10s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 21:29:46 (1779067786) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0172298 s, 178 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0359746 s, 114 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000909037 s, 3.4 MB/s PASS 10b (8s) == sanityn test 11: execution of file opened for write should return error ============================================================== 21:29:54 (1779067794) striped dir -i1 -c2 -H crush2 /mnt/lustre/d11 multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8189 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 300: /mnt/lustre2/d11/f: Text file busy PASS 11 (9s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 21:30:03 (1779067803) warning: '-runtime' deprecated, use '-t runtime' instead - link 283 (time 1779067816.23 total 10.04 last 28.19) - link 581 (time 1779067826.26 total 20.07 last 29.71) - link 859 (time 1779067836.27 total 30.08 last 27.79) - link 1138 (time 1779067846.28 total 40.09 last 27.86) - link 1396 (time 1779067856.32 total 50.13 last 25.70) - link 1677 (time 1779067866.34 total 60.15 last 28.03) - link 1989 (time 1779067876.37 total 70.18 last 31.11) - link 2272 (time 1779067886.40 total 80.21 last 28.21) - link 2569 (time 1779067896.44 total 90.25 last 29.58) total: 2836 link in 100.00 seconds: 28.36 ops/second using seed 1815704569 running for 100 seconds - stat 6690 (time 1779067818 ; total 11 ; last 11) - stat 10000 (time 1779067822 ; total 15 ; last 4) - stat 16478 (time 1779067833 ; total 26 ; last 11) - stat 20000 (time 1779067838 ; total 31 ; last 5) - stat 27691 (time 1779067849 ; total 42 ; last 11) - stat 30000 (time 1779067852 ; total 45 ; last 3) - stat 34599 (time 1779067863 ; total 56 ; last 11) - stat 38298 (time 1779067874 ; total 67 ; last 11) - stat 40000 (time 1779067878 ; total 71 ; last 4) - stat 43012 (time 1779067889 ; total 82 ; last 11) - stat 46552 (time 1779067900 ; total 93 ; last 11) total: 48440 stats in 100 seconds: 484.399994 stats/second - unlinked 0 (time 1779067857 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile2836) error: No such file or directory total: 2835 unlinks in 79 seconds: 35.886074 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (22676) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (22679) - No such process PASS 12 (140s) == sanityn test 13: test directory page revocation ======= 21:32:23 (1779067943) striped dir -i1 -c2 -H crush /mnt/lustre/d13 f13 f13 PASS 13 (11s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 21:32:34 (1779067954) striped dir -i0 -c2 -H all_char /mnt/lustre/d14aa.sanityn multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.8189 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 335: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 23891 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (10s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 21:32:45 (1779067965) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d14ab.sanityn open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 24482 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (10s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 21:32:54 (1779067974) striped dir -i0 -c2 -H all_char /mnt/lustre/d14b.sanityn truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 350: 25073 Terminated $DIR1/$tdir/sleep 60 PASS 14b (8s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 21:33:02 (1779067982) striped dir -i0 -c2 -H crush /mnt/lustre/d14c.sanityn cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 362: 25667 Terminated $DIR1/$tdir/sleep 60 PASS 14c (8s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 21:33:10 (1779067990) striped dir -i0 -c2 -H crush2 /mnt/lustre/d14d.sanityn chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 373: 26263 Terminated $DIR1/$tdir/sleep 60 PASS 14d (10s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 21:33:20 (1779068000) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky Waiting for MDT destroys to complete 0 STRIPECOUNT=2 ORIGFREE=7207404 MAXFREE=2097152000 0 lustre-OST0000 avl=0 grnt=779 diff=-779 limit=46331 FULL lustre-OST0001 avl=336 grnt=311 diff=25 limit=46331 FULL ERROR: dd blocks 7198157 != file size 7198176 LOG LOG2 file /tmp/oos2.log:dd: error writing '/mnt/lustre/oosfile': No space left on device /tmp/oos2.log:3580317+0 records in /tmp/oos2.log:3580316+0 records out /tmp/oos2.log:3666243584 bytes (3.7 GB, 3.4 GiB) copied, 967.428 s, 3.8 MB/s /tmp/oos2.log2:dd: error writing '/mnt/lustre2/oosfile2': No space left on device /tmp/oos2.log2:3617842+0 records in /tmp/oos2.log2:3617841+0 records out /tmp/oos2.log2:3704669184 bytes (3.7 GB, 3.5 GiB) copied, 967.474 s, 3.8 MB/s Waiting for MDT destroys to complete Waiting for MDT destroys to complete PASS 15 (1039s) == sanityn test 16a: 2500 iterations of dual-mount fsx === 21:50:39 (1779069039) keep default fallocate mode: 0 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.2925 s, 28.7 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.159574 s, 52.6 MB/s Chance of close/open is 1 in 50 Seed set to 9043 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn skipping zero size read truncating to largest ever: 0x3ee2 fallocating to largest ever: 0xa00000 000100[1] 1779069047.785080 read 0x1f17a6 thru 0x1fe49d (0xccf8 bytes) 000200[1] 1779069051.307408 read 0x1556d7 thru 0x164750 (0xf07a bytes) 000300[1] 1779069055.304629 trunc from 0x6b15e5 to 0x6b807a (0x6a96 bytes) 000400[1] 1779069058.519328 write 0x8f1c5e thru 0x8f5a9f (0x3e42 bytes) 000500[0] 1779069062.095566 mapwrite 0x3cf359 thru 0x3db9ba (0xc662 bytes) 000600[1] 1779069066.313666 mapwrite 0x0e2fa0 thru 0x0e648d (0x34ee bytes) 000700[0] 1779069069.985757 punch from 0x69d7fb to 0x6a189f (0x40a5 bytes) 000800[1] 1779069074.022251 write 0x7b3125 thru 0x7c3072 (0xff4e bytes) 000900[0] 1779069078.067563 trunc from 0x79b33d to 0x25916f (0x5421cd bytes) 001000[0] 1779069081.709246 trunc from 0x47aa53 to 0x1a1dfd (0x2d8c55 bytes) 001100[1] 1779069086.587602 punch from 0x1445e6 to 0x14fd6e (0xb789 bytes) 001200[0] 1779069092.141784 fallocate 0x6cbf23 thru 0x6d856a (0xc648 bytes) 001300[0] 1779069097.289532 read 0x6523d4 thru 0x660d35 (0xe962 bytes) 001400[0] 1779069102.170857 trunc from 0x64b6cf to 0x969b32 (0x31e464 bytes) 001600[0] 1779069110.301854 mapwrite 0x336217 thru 0x339309 (0x30f3 bytes) 001700[1] 1779069114.621086 trunc from 0x0174f3 to 0x02229e (0xadac bytes) 001900[0] 1779069122.030906 mapread 0x7e9df1 thru 0x7f425c (0xa46c bytes) 002000[1] 1779069125.304776 punch from 0x0a2bde to 0x0aa3f8 (0x781b bytes) 002100[0] 1779069128.741100 punch from 0x528058 to 0x5293c7 (0x1370 bytes) 002200[1] 1779069133.158764 mapread 0x5ffcd1 thru 0x60e745 (0xea75 bytes) 002300[1] 1779069138.300975 punch from 0x82a0cb to 0x8390cb (0xf001 bytes) 002400[1] 1779069142.339454 fallocate 0x83cfac thru 0x849922 (0xc977 bytes) 002500[0] 1779069146.593556 fallocate 0x9de043 thru 0x9e602a (0x7fe8 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 9146 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn truncating to largest ever: 0x50fe95 fallocating to largest ever: 0x5ca74b truncating to largest ever: 0x956da2 fallocating to largest ever: 0x97f6c0 000100[0] 1779069149.977000 trunc from 0x7c38f7 to 0x331df0 (0x491b06 bytes) fallocating to largest ever: 0x9a27ab fallocating to largest ever: 0x9b2ecc truncating to largest ever: 0x9e6c04 000200[0] 1779069152.600205 write_OD 0x693000 thru 0x699fff (0x7000 bytes) truncating to largest ever: 0x9f8209 000300[0] 1779069154.719985 fallocate 0x2e26b9 thru 0x2ecb0f (0xa457 bytes) 000400[1] 1779069157.193021 mapwrite 0x80e000 thru 0x81b7f7 (0xd7f8 bytes) 000500[0] 1779069159.924337 punch from 0x48b1d4 to 0x493212 (0x803f bytes) fallocating to largest ever: 0x9f9712 truncating to largest ever: 0x9fb8b2 000600[1] 1779069162.768409 punch from 0x08a377 to 0x0970c5 (0xcd4f bytes) 000800[1] 1779069168.491832 trunc from 0x879ecf to 0x149adf (0x7303ef bytes) 000900[1] 1779069171.571451 punch from 0x1bd6a0 to 0x1c5484 (0x7de5 bytes) 001100[0] 1779069177.223211 write_OD 0x415000 thru 0x41efff (0xa000 bytes) 001500[1] 1779069189.018070 mapread 0x401000 thru 0x409c1e (0x8c1f bytes) 001600[1] 1779069192.734929 mapread 0x06c000 thru 0x06f3df (0x33e0 bytes) 001700[1] 1779069195.840920 read 0x035000 thru 0x042155 (0xd156 bytes) 001800[1] 1779069199.693341 trunc from 0x0ccb9c to 0x0d105b (0x44c0 bytes) 001900[0] 1779069203.025007 mapread 0x597000 thru 0x59f18d (0x818e bytes) 002000[0] 1779069206.581963 read_OD 0x55d000 thru 0x55ffff (0x3000 bytes) 002100[1] 1779069209.814996 fallocate 0x4209d2 thru 0x427115 (0x6744 bytes) 002200[1] 1779069212.814633 read_OD 0x242000 thru 0x246fff (0x5000 bytes) 002300[0] 1779069215.912367 trunc from 0x10291d to 0x1091c8 (0x68ac bytes) 002400[0] 1779069219.386611 read 0x16d000 thru 0x171ac3 (0x4ac4 bytes) All operations completed A-OK! PASS 16a (187s) == sanityn test 16b: 2500 iterations of dual-mount fsx at small size ========================================================== 21:53:46 (1779069226) keep default fallocate mode: 0 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.262929 s, 31.9 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.207375 s, 40.5 MB/s Chance of close/open is 1 in 50 Seed set to 9229 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn skipping zero length punch hole truncating to largest ever: 0x5187d zero_range to largest ever: 0x1b8975 zero_range to largest ever: 0x74d1db truncating to largest ever: 0x778493 truncating to largest ever: 0x9ac34a fallocating to largest ever: 0x9c3227 fallocating to largest ever: 0x9fb1a1 000200[0] 1779069237.249834 mapread 0x071843 thru 0x071bb4 (0x372 bytes) 000300[0] 1779069240.662999 fallocate 0x88cad2 thru 0x88de7e (0x13ad bytes) 000400[0] 1779069243.950564 fallocate 0x00f564 thru 0x010e0c (0x18a9 bytes) 000500[0] 1779069247.229329 trunc from 0x3dbe75 to 0x3ddd86 (0x1f12 bytes) 000900[0] 1779069261.381929 mapwrite 0x12eac7 thru 0x1300cb (0x1605 bytes) 001000[0] 1779069264.746996 trunc from 0x40d61b to 0x40ed25 (0x170b bytes) 001100[0] 1779069267.490409 trunc from 0x9f7b1b to 0x59b863 (0x45c2b7 bytes) 001200[0] 1779069270.316017 read 0x003d1c thru 0x004ae3 (0xdc8 bytes) 001300[0] 1779069273.475622 trunc from 0x8d6095 to 0x98dca4 (0xb7c10 bytes) 001400[0] 1779069277.049780 trunc from 0x7c4d8f to 0x497425 (0x32d969 bytes) fallocating to largest ever: 0x9fdf61 001500[0] 1779069280.927200 write 0x79b51a thru 0x79c74f (0x1236 bytes) 001600[1] 1779069283.857932 mapread 0x0f94f9 thru 0x0fa081 (0xb89 bytes) 001800[1] 1779069290.842504 trunc from 0x60ccb1 to 0x603621 (0x968f bytes) 001900[1] 1779069294.779944 mapread 0x2fb8c6 thru 0x2fbd53 (0x48e bytes) 002000[1] 1779069298.251435 fallocate 0x4e73ab thru 0x4e8703 (0x1359 bytes) 002100[1] 1779069301.331143 mapwrite 0x73cbf0 thru 0x73ddc5 (0x11d6 bytes) 002200[0] 1779069304.703811 fallocate 0x579c55 thru 0x57b493 (0x183f bytes) 002400[1] 1779069311.053556 trunc from 0x7ad819 to 0x995073 (0x1e785b bytes) 002500[0] 1779069314.585494 read 0x2d4751 thru 0x2d5b1d (0x13cd bytes) All operations completed A-OK! PASS 16b (93s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 21:55:19 (1779069319) keep default fallocate mode: 0 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.183073 s, 45.8 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.194363 s, 43.2 MB/s Waiting for MDT destroys to complete Chance of close/open is 1 in 50 Seed set to 9338 fd 0: /mnt/lustre/f16c.sanityn fd 1: /mnt/lustre2/f16c.sanityn fallocating to largest ever: 0x8f6e9 fallocating to largest ever: 0x54fa47 zero_range to largest ever: 0x5c181b fallocating to largest ever: 0x7e74dd truncating to largest ever: 0x906534 000100[0] 1779069342.787755 trunc from 0x06980f to 0x072447 (0x8c39 bytes) truncating to largest ever: 0x973f1f fallocating to largest ever: 0xa00000 000200[1] 1779069346.248732 read 0x38dfac thru 0x3971ce (0x9223 bytes) 000400[1] 1779069353.405645 mapread 0x147e3c thru 0x147ef9 (0xbe bytes) 000500[1] 1779069356.912374 write 0x16fbc5 thru 0x17300d (0x3449 bytes) 000600[0] 1779069360.449663 trunc from 0x92085e to 0x891776 (0x8f0e7 bytes) 000800[0] 1779069368.821891 fallocate 0x5f6e6e thru 0x5ff38d (0x8520 bytes) 000900[1] 1779069372.355474 mapwrite 0x9f04e9 thru 0x9f4f1e (0x4a36 bytes) 001000[1] 1779069376.097013 mapread 0x1cae66 thru 0x1d9f04 (0xf09f bytes) 001100[0] 1779069379.407702 mapwrite 0x9e2e21 thru 0x9e8106 (0x52e6 bytes) 001200[0] 1779069382.724145 write 0x16e3cf thru 0x17bb3b (0xd76d bytes) 001300[1] 1779069386.081428 mapread 0x0cc150 thru 0x0d888b (0xc73c bytes) 001400[0] 1779069389.772952 mapread 0x362daf thru 0x367120 (0x4372 bytes) 001500[0] 1779069392.803165 fallocate 0x34981d thru 0x353240 (0x9a24 bytes) 001600[0] 1779069396.153587 mapwrite 0x13cca4 thru 0x13dfb3 (0x1310 bytes) 001800[0] 1779069402.223332 fallocate 0x51dd6b thru 0x52a8a2 (0xcb38 bytes) 001900[0] 1779069405.128678 fallocate 0x34b2ca thru 0x351963 (0x669a bytes) 002000[1] 1779069407.985209 fallocate 0x9b5b79 thru 0x9c011b (0xa5a3 bytes) 002100[1] 1779069410.963855 write 0x2c15a9 thru 0x2cd3ad (0xbe05 bytes) 002200[1] 1779069414.336851 write 0x8b9fa7 thru 0x8c6d7e (0xcdd8 bytes) 002300[1] 1779069417.670413 mapread 0x02e28d thru 0x0332a1 (0x5015 bytes) 002400[0] 1779069422.037390 write 0x477aa5 thru 0x487127 (0xf683 bytes) All operations completed A-OK! PASS 16c (111s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 21:57:10 (1779069430) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.135559 s, 61.9 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.118649 s, 70.7 MB/s ldlm.namespaces.MGC192.168.201.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffffa0e708efa800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffffa0e71017f800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffffa0e708efa800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffffa0e71017f800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffffa0e708efa800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffffa0e71017f800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffffa0e708efa800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffffa0e71017f800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.09437 s, 20.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.50384 s, 41.9 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.58455 s, 40.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.67255 s, 18.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.65028 s, 28.7 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.44127 s, 30.5 MB/s PASS 16d (32s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 21:57:42 (1779069462) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.256826 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317307 s, 33.0 MB/s PASS 16e (6s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 21:57:48 (1779069468) TIMEOUT PASS 16f (26s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 21:58:14 (1779069494) TIMEOUT PASS 16g (25s) == sanityn test 16h: mmap read after truncate file ======= 21:58:39 (1779069519) striped dir -i0 -c2 -H all_char /mnt/lustre/d16h.sanityn ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (5s) == sanityn test 16i: read after truncate file ============ 21:58:44 (1779069524) striped dir -i0 -c2 -H crush /mnt/lustre/d16i.sanityn ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (5s) == sanityn test 16j: race dio with buffered i/o ========== 21:58:49 (1779069529) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.64131 s, 40.9 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.84467 s, 31.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.08782 s, 24.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.65076 s, 40.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.973834 s, 26.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.761637 s, 34.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.2412 s, 21.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.11594 s, 23.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.25691 s, 20.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.29233 s, 20.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.47249 s, 17.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.05979 s, 24.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.11914 s, 23.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.48738 s, 17.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.54441 s, 17.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.27967 s, 20.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.36937 s, 19.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.76867 s, 14.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.83122 s, 14.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (22s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 21:59:11 (1779069551) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fallocating to largest ever: 0x3428364 skipping zero size read fallocating to largest ever: 0x3428364 fallocating to largest ever: 0x3428364 fallocating to largest ever: 0x3428364 skipping zero size read skipping zero size read skipping zero size read fallocating to largest ever: 0x3428364 skipping zero size read truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db fallocating to largest ever: 0x3ddbbf6 fallocating to largest ever: 0x3ddbbf6 fallocating to largest ever: 0x3ddbbf6 fallocating to largest ever: 0x3ddbbf6 fallocating to largest ever: 0x3ddbbf6 fallocating to largest ever: 0x3fa9cee fallocating to largest ever: 0x3fa9cee fallocating to largest ever: 0x3fa9cee fallocating to largest ever: 0x3fa9cee fallocating to largest ever: 0x3fa9cee truncating to largest ever: 0x3fb80f5 truncating to largest ever: 0x3fb80f5 truncating to largest ever: 0x3fb80f5 truncating to largest ever: 0x3fb80f5 truncating to largest ever: 0x3fb80f5 All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Sun May 17 21:59:41 EDT 2026 + fsx 2 done: rc=0 Sun May 17 21:59:41 EDT 2026 + fsx 3 done: rc=0 Sun May 17 21:59:41 EDT 2026 + fsx 4 done: rc=0 Sun May 17 21:59:41 EDT 2026 + fsx 5 done: rc=0 Sun May 17 21:59:41 EDT 2026 PASS 16k (36s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 21:59:47 (1779069587) fail_loc=0x8000030a PASS 17 (9s) == sanityn test 18: mmap sanity check =========================================================================================== 21:59:56 (1779069596) mmap test1: basic mmap operation (PASS, 0.11144s) mmap test2: MAP_PRIVATE not write back (PASS, 0.071489s) mmap test3: concurrent mmap ops on two nodes (PASS, 13.426s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.3959s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.16259s) mmap test6: check mmap write/read content on two nodes (PASS, 0.17227s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 0.86706s) mmap test11: multi-thread shared mmap (PASS, 0.95618s) mmap test10: mtime not change for readonly mmap access (PASS, 1.0831s) PASS 18 (28s) == sanityn test 19: test concurrent uncached read races ========================================================================= 22:00:24 (1779069624) 32+0 records in 32+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.31696 s, 52.9 MB/s loop 5 loop 10 loop 15 loop 20 PASS 19 (25s) == sanityn test 20: test extra readahead page left in cache ============================================================== 22:00:49 (1779069649) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d20.sanityn PASS 20 (6s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 22:00:55 (1779069655) striped dir -i1 -c2 -H crush /mnt/lustre/d21.sanityn rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (5s) == sanityn test 23: others should see updated atime while another read============================================================== 22:01:00 (1779069660) now is 1779069663 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.8189 new atime is 1779069724 PASS 23 (67s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 22:02:07 (1779069727) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2036 1269268 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1792 1269512 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 30620 3576400 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 7972 3599048 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 38592 7175448 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2036 1269268 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1792 1269512 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 30620 3576400 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 7972 3599048 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 38592 7175448 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1000.0K 351 999.7K 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1000.0K 281 999.7K 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 256.0K 402 255.6K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 256.0K 371 255.6K 1% /mnt/lustre[OST:1] filesystem_summary: 511.9K 632 511.2K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1000.0K 351 999.7K 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1000.0K 281 999.7K 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 256.0K 402 255.6K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 256.0K 371 255.6K 1% /mnt/lustre2[OST:1] filesystem_summary: 511.9K 632 511.2K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 1.3G 2.0M 1.2G 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1.3G 1.8M 1.2G 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3.7G 29.9M 3.4G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.7G 7.8M 3.4G 1% /mnt/lustre[OST:1] filesystem_summary: 7.3G 37.7M 6.8G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1024000 351 1023649 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1024000 281 1023719 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 262144 402 261742 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 262144 371 261773 1% /mnt/lustre2[OST:1] filesystem_summary: 524147 632 523515 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2036 1269268 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1792 1269512 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 30620 3576400 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 7972 3599048 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 38592 7175448 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1000.0K 351 999.7K 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1000.0K 281 999.7K 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 256.0K 402 255.6K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 256.0K 371 255.6K 1% /mnt/lustre2[OST:1] filesystem_summary: 511.9K 632 511.2K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1024000 351 1023649 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1024000 281 1023719 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 262144 402 261742 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 262144 371 261773 1% /mnt/lustre[OST:1] filesystem_summary: 524147 632 523515 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1024000 351 1023649 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1024000 281 1023719 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 262144 402 261742 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 262144 371 261773 1% /mnt/lustre2[OST:1] filesystem_summary: 524147 632 523515 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2036 1269268 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1792 1269512 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 30620 3576400 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 7972 3599048 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 38592 7175448 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2036 1269268 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1792 1269512 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 30620 3576400 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 7972 3599048 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 38592 7175448 1% /mnt/lustre2 PASS 24a (5s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 22:02:12 (1779069732) PASS 24b (4s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 22:02:16 (1779069736) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (5s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 22:02:21 (1779069741) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] Can't lstat /mnt/lustre2/d25b.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] Can't lstat /mnt/lustre2/d25b.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] PASS 25b (6s) == sanityn test 26a: allow mtime to get older ============ 22:02:27 (1779069747) utime: good mknod times 1779069747 <= 1779069748 <= 1779069748 for /mnt/lustre/f26a utime: good mknod times 1779069747 <= 1779069748 <= 1779069748 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (5s) == sanityn test 26b: sync mtime between ost and mds ====== 22:02:32 (1779069752) PASS 26b (6s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 22:02:38 (1779069758) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8189 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205289279610 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-05-17 22:02:39.000000000 -0400 Birth: 2026-05-17 22:02:39.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205289279610 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-05-17 22:02:39.000000000 -0400 Birth: 2026-05-17 22:02:39.000000000 -0400 PASS 26c (5s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 22:02:44 (1779069764) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d30.sanityn 0 PASS 30 (6s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 22:02:50 (1779069770) striped dir -i1 -c2 -H crush2 /mnt/lustre/d31a.sanityn fail_loc=0x314 PASS 31a (6s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 22:02:56 (1779069776) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg135-server mds-ost sync done. Waiting for MDT destroys to complete striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d31b.sanityn fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.135@tcp:/lustre 7666232 38592 7174272 1% /mnt/lustre2 PASS 31b (12s) == sanityn test 31r: open-rename(replace) race =========== 22:03:08 (1779069788) -rw-r--r-- 1 root root 0 May 17 22:03 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (7s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 22:03:15 (1779069795) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1207: 47563 Terminated tail -f $tfile PASS 31s (6s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 22:03:21 (1779069801) PASS 31t (5s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 22:03:28 (1779069808) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (2s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 22:03:30 (1779069810) Stopping /mnt/lustre-mds1 (opts:) on oleg135-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg135-server: oleg135-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-MDT0000 create file on 2nd stripe under striped directory /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 2 /mnt/lustre/d33c.sanityn/striped touch /mnt/lustre2/d33c.sanityn/striped/subfile SoLC count 2 Below operations shouldn't trigger Solc: create file on 2nd stripe after setattr chmod 777 /mnt/lustre/d33c.sanityn/striped; touch /mnt/lustre2/d33c.sanityn/striped/subfile create remote dir and local dir /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33c.sanityn/remote create local dir after remote dir creation transaction commit /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33c.sanityn/remote.3; do_facet mds2 /home/green/git/lustre-release/lustre/utils/lctl set_param -n osd*.*MDT0001.force_sync 1; mkdir /mnt/lustre/d33c.sanityn/local.3 Solc count 0 PASS 33c (31s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 22:04:01 (1779069841) remote directory create /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre2/d33d.sanityn/subdir CoS count 2 cross-MDT rename mkdir /mnt/lustre/d33d.sanityn; /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn/d1; /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33d.sanityn/d2; touch /mnt/lustre/d33d.sanityn/d1/src mv /mnt/lustre2/d33d.sanityn/d1/src /mnt/lustre2/d33d.sanityn/d2/tgt CoS count 1 migrate /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn /home/green/git/lustre-release/lustre/utils/lfs migrate -m 1 /mnt/lustre2/d33d.sanityn CoS count 2 PASS 33d (31s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 22:04:32 (1779069872) plain dir creation mkdir /mnt/lustre2/d33e.sanityn/plain open file and write echo abc > /mnt/lustre2/d33e.sanityn/f33e.sanityn append write echo dfg >> /mnt/lustre2/d33e.sanityn/f33e.sanityn setattr touch /mnt/lustre2/d33e.sanityn/f33e.sanityn file unlink rm /mnt/lustre2/d33e.sanityn/f33e.sanityn plain dir unlink rmdir /mnt/lustre2/d33e.sanityn/plain striped directory creation /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 2 /mnt/lustre2/d33e.sanityn/striped set default LMV to create striped subdir /home/green/git/lustre-release/lustre/utils/lfs setdirstripe -D -c 2 /mnt/lustre/d33e.sanityn striped subdir creation createmany -d /mnt/lustre/d33e.sanityn/subdir 100 total: 100 mkdir in 1.00 seconds: 99.65 ops/second sub file creation and write createmany -o /mnt/lustre/d33e.sanityn/subfile 100; echo abc > /mnt/lustre/d33e.sanityn/subfile1 total: 100 open/close in 1.13 seconds: 88.12 ops/second sub file append write echo dfg >> /mnt/lustre2/d33e.sanityn/subfile2 subdir setatttr touch /mnt/lustre2/d33e.sanityn/subdir1 subdir unlink unlinkmany -d /mnt/lustre/d33e.sanityn/subdir 100 - unlinked 0 (time 1779069877 ; total 0 ; last 0) total: 100 unlinks in 4 seconds: 25.000000 unlinks/second sub file unlink unlinkmany /mnt/lustre2/d33e.sanityn/subfile 100 - unlinked 0 (time 1779069881 ; total 0 ; last 0) total: 100 unlinks in 2 seconds: 50.000000 unlinks/second sub file creation follows striped dir chmod chmod 777 /mnt/lustre/d33e.sanityn/striped; touch /mnt/lustre/d33e.sanityn/striped/subfile striped directory unlink rm -rf /mnt/lustre2/d33e.sanityn/striped directory unlink rm -rf /mnt/lustre2/d33e.sanityn CoS count 0 PASS 33e (15s) == sanityn test 34: no lock timeout under IO ============= 22:04:47 (1779069887) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg135-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffffa0e708efa800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa0e708efa800.ost_server_uuid in FULL state after 0 sec oleg135-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa0e708efa800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa0e708efa800.ost_server_uuid in IDLE state after 0 sec oleg135-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffffa0e708efa800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa0e708efa800.ost_server_uuid in FULL state after 0 sec oleg135-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa0e708efa800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa0e708efa800.ost_server_uuid in IDLE state after 0 sec oleg135-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffffa0e708efa800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa0e708efa800.ost_server_uuid in IDLE state after 0 sec oleg135-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa0e708efa800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa0e708efa800.ost_server_uuid in IDLE state after 0 sec PASS 34 (97s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 22:06:24 (1779069984) striped dir -i1 -c2 -H crush /mnt/lustre/d35.sanityn fail_loc=0x80000317 Race attempt 0 Wait for 57553 57639 for 60 sec... - open/close 1027 (time 1779069998.95 total 10.00 last 102.70) - open/close 2299 (time 1779070008.96 total 20.00 last 127.14) - open/close 3665 (time 1779070018.96 total 30.01 last 136.59) total: 4000 open/close in 32.71 seconds: 122.29 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.201.135@tcp:/lustre 7.4G 38M 6.9G 1% /mnt/lustre 192.168.201.135@tcp:/lustre 7.4G 38M 6.9G 1% /mnt/lustre2 PASS 35 (68s) == sanityn test 36: handle ESTALE/open-unlink correctly == 22:07:32 (1779070052) striped dir -i0 -c2 -H crush2 /mnt/lustre/d36.sanityn Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg135-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.656558 s, 79.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8189 Waiting for MDT destroys to complete *** cycle(0) *** before(7175388) after_dd(7124188) after(7175388) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.628253 s, 83.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8189 Waiting for MDT destroys to complete *** cycle(1) *** before(7175388) after_dd(7124188) after(7175388) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.546337 s, 96.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8189 Waiting for MDT destroys to complete *** cycle(2) *** before(7175388) after_dd(7124188) after(7175388) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.522276 s, 100 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8189 Waiting for MDT destroys to complete *** cycle(3) *** before(7175388) after_dd(7124188) after(7175388) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.603213 s, 86.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8189 Waiting for MDT destroys to complete *** cycle(4) *** before(7175388) after_dd(7124188) after(7175388) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.577793 s, 90.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8189 Waiting for MDT destroys to complete *** cycle(5) *** before(7175388) after_dd(7124188) after(7175388) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.562328 s, 93.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8189 Waiting for MDT destroys to complete *** cycle(6) *** before(7175388) after_dd(7124188) after(7175388) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.620068 s, 84.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8189 Waiting for MDT destroys to complete *** cycle(7) *** before(7175388) after_dd(7124188) after(7175388) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.557061 s, 94.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8189 Waiting for MDT destroys to complete *** cycle(8) *** before(7175388) after_dd(7124188) after(7175388) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.587702 s, 89.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8189 Waiting for MDT destroys to complete *** cycle(9) *** before(7175388) after_dd(7124188) after(7175388) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.563811 s, 93.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8189 Waiting for MDT destroys to complete *** cycle(10) *** before(7175388) after_dd(7124188) after(7175388) PASS 36 (185s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 22:10:37 (1779070237) striped dir -i1 -c2 -H all_char /mnt/lustre/d37.sanityn multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.8189 - create 5893 (time 1779070249.40 total 10.00 last 589.29) total: 10000 create in 17.19 seconds: 581.84 ops/second PASS 37 (22s) == sanityn test 39a: file mtime does not change after rename ========================================================== 22:10:59 (1779070259) repeat after cancel_lru_locks PASS 39a (3s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 22:11:02 (1779070262) repeat after cancel_lru_locks PASS 39b (4s) == sanityn test 39c: check truncate mtime update ================================================================================ 22:11:06 (1779070266) repeat after cancel_lru_locks PASS 39c (4s) == sanityn test 39d: sync write should update mtime ====== 22:11:10 (1779070270) fail_loc=0x411 fail_loc=0 PASS 39d (2s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 22:11:13 (1779070273) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (9s) == sanityn test 40c: pdirops: link and others ======================================================================== 22:11:22 (1779070282) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=7 No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (8s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 22:11:31 (1779070291) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (9s) == sanityn test 40e: pdirops: rename and others ======================================================================== 22:11:39 (1779070299) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (9s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 22:11:48 (1779070308) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (6s) == sanityn test 41b: pdirops: create vs create ======================================================================== 22:11:54 (1779070314) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 open: File exists Conflict PASS 41b (6s) == sanityn test 41c: pdirops: create vs link ======================================================================== 22:12:00 (1779070320) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (7s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 22:12:07 (1779070327) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=4 Conflict PASS 41d (5s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 22:12:12 (1779070332) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 Conflict PASS 41e (6s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 22:12:18 (1779070338) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=5 Conflict PASS 41f (6s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 22:12:24 (1779070344) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=6 Conflict PASS 41g (6s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 22:12:30 (1779070350) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 Conflict PASS 41h (6s) == sanityn test 41i: reint_open: create vs create ======== 22:12:36 (1779070356) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (744s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 22:25:00 (1779071100) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (5s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 22:25:05 (1779071105) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=1 open: File exists Conflict PASS 42b (6s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 22:25:11 (1779071111) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (5s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 22:25:16 (1779071116) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 Conflict PASS 42d (5s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 22:25:21 (1779071121) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (5s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 22:25:26 (1779071126) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 Conflict PASS 42f (6s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 22:25:32 (1779071132) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=4 Conflict PASS 42g (5s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 22:25:37 (1779071137) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 Conflict PASS 42h (5s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 22:25:42 (1779071142) PASS 43a (24s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 22:26:06 (1779071166) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=33 Conflict PASS 43b (5s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 22:26:11 (1779071171) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 43c (5s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 22:26:16 (1779071176) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (6s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 22:26:22 (1779071182) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 43e (5s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 22:26:27 (1779071187) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (5s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 22:26:32 (1779071192) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=4 stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (5s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 22:26:37 (1779071197) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 Conflict PASS 43h (5s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 22:26:42 (1779071202) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 Conflict PASS 43i (5s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 22:26:48 (1779071208) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (43s) == sanityn test 43k: unlink vs create ==================== 22:27:30 (1779071250) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (458s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 22:35:08 (1779071708) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Sun May 17 22:35:11 EDT 2026 PASS 44a (5s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 22:35:13 (1779071713) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=7 open: File exists Conflict PASS 44b (5s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 22:35:18 (1779071718) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=5 link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (5s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 22:35:23 (1779071723) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 Conflict PASS 44d (4s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 22:35:28 (1779071728) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 44e (5s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 22:35:32 (1779071732) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 44f (5s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 22:35:37 (1779071737) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=12 Conflict PASS 44g (5s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 22:35:42 (1779071742) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=6 Conflict PASS 44h (5s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 22:35:47 (1779071747) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=5 lfs mkdir: dirstripe error on '/mnt/lustre2/f44i.sanityn': stripe already set lfs setdirstripe: cannot create dir '/mnt/lustre2/f44i.sanityn': File exists Conflict PASS 44i (5s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 22:35:52 (1779071752) PASS 45a (39s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 22:36:31 (1779071791) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=56 Conflict PASS 45b (5s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 22:36:36 (1779071796) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=9 Conflict PASS 45c (5s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 22:36:41 (1779071801) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=12 rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (6s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 22:36:47 (1779071807) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=6 Conflict PASS 45e (5s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 22:36:52 (1779071812) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=12 mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (5s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 22:36:57 (1779071817) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=6 stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (6s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 22:37:03 (1779071823) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45h (5s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 22:37:08 (1779071828) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=7 Conflict PASS 45i (5s) == sanityn test 45j: read vs rename ====================== 22:37:13 (1779071833) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (462s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 22:44:55 (1779072295) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (5s) == sanityn test 46b: pdirops: link vs create ======================================================================== 22:45:00 (1779072300) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 open: File exists Conflict PASS 46b (5s) == sanityn test 46c: pdirops: link vs link ======================================================================== 22:45:05 (1779072305) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (5s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 22:45:10 (1779072310) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 Conflict PASS 46d (6s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 22:45:16 (1779072316) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 46e (5s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 22:45:21 (1779072321) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 Conflict PASS 46f (5s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 22:45:26 (1779072326) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 Conflict PASS 46g (5s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 22:45:31 (1779072331) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 Conflict PASS 46h (5s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 22:45:36 (1779072336) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 lfs mkdir: dirstripe error on '/mnt/lustre2/f46i.sanityn': stripe already set lfs setdirstripe: cannot create dir '/mnt/lustre2/f46i.sanityn': File exists Conflict PASS 46i (5s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 22:45:41 (1779072341) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 mkdir: cannot create directory '/mnt/lustre2/f47a.sanityn': File exists Conflict PASS 47a (6s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 22:45:47 (1779072347) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=1 open: File exists Conflict PASS 47b (6s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 22:45:53 (1779072353) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 ldlm.namespaces.mdt-lustre-MDT0001_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f47c.sanityn' to '/mnt/lustre2/f47c.sanityn-2': File exists Conflict PASS 47c (5s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 22:45:58 (1779072358) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 Conflict PASS 47d (5s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 22:46:03 (1779072363) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=2 mv: cannot overwrite directory '/mnt/lustre2/f47e.sanityn' with non-directory Conflict PASS 47e (5s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 22:46:08 (1779072368) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 Conflict PASS 47f (5s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 22:46:13 (1779072373) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 47g (6s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 22:46:19 (1779072379) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.00384549 s, 2.7 MB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (7s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 22:46:26 (1779072386) 0+1 records in 0+1 records out 158 bytes copied, 0.00467367 s, 33.8 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3510: kill: (284920) - No such process PASS 51a (4s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 22:46:30 (1779072390) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00396143 s, 258 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171818 s, 61.0 MB/s 3145728 PASS 51b (14s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 22:46:44 (1779072404) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00340809 s, 300 kB/s PASS 51c (6s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 22:46:51 (1779072411) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118357 s, 88.6 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (4s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 22:46:54 (1779072414) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 1317 0x525 0x2c0000401 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 1317 0x525 0x2c0000401 read lease(1) has applied. PASS 51e (4s) == sanityn test 54: rename locking ======================= 22:46:58 (1779072418) llite.lustre-ffffa0e708efa800.xattr_cache=1 llite.lustre-ffffa0e71017f800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205289281281 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-17 22:46:58.000000000 -0400 Modify: 2026-05-17 22:46:58.000000000 -0400 Change: 2026-05-17 22:46:58.000000000 -0400 Birth: 2026-05-17 22:46:58.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798762672 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-17 22:47:04.000000000 -0400 Modify: 2026-05-17 22:47:04.000000000 -0400 Change: 2026-05-17 22:47:04.000000000 -0400 Birth: 2026-05-17 22:47:03.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205289281284 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-17 22:47:09.000000000 -0400 Modify: 2026-05-17 22:47:09.000000000 -0400 Change: 2026-05-17 22:47:09.000000000 -0400 Birth: 2026-05-17 22:47:09.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798762673 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-17 22:47:09.000000000 -0400 Modify: 2026-05-17 22:47:10.000000000 -0400 Change: 2026-05-17 22:47:10.000000000 -0400 Birth: 2026-05-17 22:47:09.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205289281285 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-17 22:47:15.000000000 -0400 Modify: 2026-05-17 22:47:15.000000000 -0400 Change: 2026-05-17 22:47:15.000000000 -0400 Birth: 2026-05-17 22:47:14.000000000 -0400 File: /mnt/lustre/d1/d2 Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798762675 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-17 22:47:15.000000000 -0400 Modify: 2026-05-17 22:47:15.000000000 -0400 Change: 2026-05-17 22:47:15.000000000 -0400 Birth: 2026-05-17 22:47:14.000000000 -0400 PASS 54 (24s) == sanityn test 55a: rename vs unlink target dir ========= 22:47:22 (1779072442) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (7s) == sanityn test 55b: rename vs unlink source dir ========= 22:47:29 (1779072449) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (8s) == sanityn test 55c: rename vs unlink orphan target dir == 22:47:37 (1779072457) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (12s) == sanityn test 55d: rename file vs link ================= 22:47:49 (1779072469) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (10s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 22:47:59 (1779072479) fail_loc=0x155 We survived after AB/BA race test fail_loc=0x0 PASS 55e (12s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 22:48:11 (1779072491) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (13s) == sanityn test 55g: rename: race with trylock =========== 22:48:24 (1779072504) mdt.lustre-MDT0000.enable_rename_trylock=1 mdt.lustre-MDT0001.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (14s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 22:48:38 (1779072518) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1779072519 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 15.2598 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (6s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 22:48:44 (1779072524) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1779072526 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 5.30757 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (27s) == sanityn test 60: Verify data_version behaviour ======== 22:49:12 (1779072552) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60.sanityn PASS 60 (2s) == sanityn test 70a: cd directory && rm directory ======== 22:49:14 (1779072554) PASS 70a (2s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 22:49:16 (1779072556) PASS 70b (3s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 22:49:19 (1779072559) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00317398 s, 20.6 MB/s 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00268324 s, 24.4 MB/s File: /mnt/lustre2/f71a.sanityn Size: 262144 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205289281328 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-17 22:49:19.000000000 -0400 Modify: 2026-05-17 22:49:19.000000000 -0400 Change: 2026-05-17 22:49:19.000000000 -0400 Birth: 2026-05-17 22:49:19.000000000 -0400 21690 extent 0 in offset 65536, length 65536 flags (0x80000000): NET extent 1 in offset 196608, length 65536 flags (0x80000001): LAST NET No unwritten extents, extents number 2, sum of lengths 131072, expected sum 131072 21691 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00214639 s, 30.5 MB/s 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00246094 s, 26.6 MB/s File: /mnt/lustre/f71a.sanityn Size: 262144 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205289281328 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-17 22:49:19.000000000 -0400 Modify: 2026-05-17 22:49:19.000000000 -0400 Change: 2026-05-17 22:49:19.000000000 -0400 Birth: 2026-05-17 22:49:19.000000000 -0400 21692 extent 0 in offset 65536, length 65536 flags (0x80000000): NET extent 1 in offset 196608, length 65536 flags (0x80000001): LAST NET No unwritten extents, extents number 2, sum of lengths 131072, expected sum 131072 21691 PASS 71a (2s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 22:49:21 (1779072561) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00304306 s, 21.5 MB/s extent 0 in offset 0, length 65536 flags (0x80000001): LAST NET No unwritten extents, extents number 1, sum of lengths 65536, expected sum 65536 PASS 71b (2s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 22:49:23 (1779072563) filefrag 1.47.3-wc2 (11-Nov-2025) check last flag for file with 291 extents 290: 296960.. 296963: 378880.. 378883: 4: 0001: last,net,eof check last flag for file with 292 extents 291: 297984.. 297987: 371712.. 371715: 4: 0000: last,net,eof check last flag for file with 293 extents 292: 299008.. 299011: 716800.. 716803: 4: 0001: last,net,eof PASS 71c (8s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 22:49:32 (1779072572) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00516643 s, 12.7 MB/s 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00218578 s, 30.0 MB/s 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00193801 s, 33.8 MB/s 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0133483 s, 4.9 MB/s disk usage: 262144 /mnt/lustre/f71d.sanityn file size: 458752 /mnt/lustre/f71d.sanityn extent 0 in offset 0, length 65536 flags (0x80000000): NET extent 1 in offset 131072, length 65536 flags (0x80000000): NET extent 2 in offset 262144, length 65536 flags (0x80000000): NET extent 3 in offset 393216, length 65536 flags (0x80000001): LAST NET No unwritten extents, extents number 4, sum of lengths 262144, expected sum 262144 PASS 71d (13s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 22:49:45 (1779072585) llite.lustre-ffffa0e708efa800.xattr_cache=1 llite.lustre-ffffa0e71017f800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (2s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 22:49:47 (1779072587) llite.lustre-ffffa0e708efa800.xattr_cache=1 llite.lustre-ffffa0e71017f800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (2s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 22:49:50 (1779072590) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00262591 s, 390 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0031096 s, 329 kB/s 300413: get lock2 300415: get lock1 300413: done 300413: get lock1 300415: done 300415: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 300413: cannot get lock1: Resource deadlock avoided fcntl cmd 7 failed: Resource deadlock avoided 300415: cannot get lock2: Resource deadlock avoided 300415: exit rc=0 300413: done 300413: exit rc=0 PASS 74 (6s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 22:49:56 (1779072596) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0227561 s, 92.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130522 s, 80.3 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4193: 301003 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (7s) == sanityn test 76: Verify MDT open_files listing ======== 22:50:03 (1779072603) striped dir -i0 -c2 -H all_char /mnt/lustre/d76.sanityn opening files: ................ closing files: ................ PASS 76 (44s) == sanityn test 77a: check FIFO NRS policy =============== 22:50:47 (1779072647) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.167347 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516089 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529241 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075068 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520118 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688368 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128687 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128782 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119354 s, 8.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143929 s, 7.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa0e708efa800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffffa0e708efa800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128466 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120363 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147497 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128243 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161886 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130769 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153771 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158842 s, 66.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177465 s, 59.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236878 s, 44.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223637 s, 46.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237394 s, 44.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00488905 s, 214 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026333 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239658 s, 43.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361486 s, 29.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109381 s, 95.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019837 s, 52.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124857 s, 84.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185244 s, 56.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163487 s, 64.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158946 s, 66.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa0e708efa800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffffa0e708efa800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00271385 s, 386 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg135-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg135-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (3s) == sanityn test 77b: check CRR-N NRS policy ============== 22:50:50 (1779072650) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.153315 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177441 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172749 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195457 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203041 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182545 s, 5.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204816 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198511 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204407 s, 5.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20045 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199804 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217105 s, 4.8 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212724 s, 4.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20624 s, 5.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21619 s, 4.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214709 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241106 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274216 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161174 s, 65.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104457 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303715 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310119 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014607 s, 71.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187592 s, 55.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160373 s, 65.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287013 s, 36.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223859 s, 46.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375425 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241009 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247073 s, 42.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261707 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262719 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278601 s, 37.6 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg135-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg135-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.136087 s, 123 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381364 s, 27.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463431 s, 22.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149015 s, 7.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136367 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143276 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139067 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13792 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165042 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148129 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155714 s, 6.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151137 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163944 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162986 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168185 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161536 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234598 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255927 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238011 s, 44.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181557 s, 57.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188557 s, 55.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226061 s, 46.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189759 s, 55.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256489 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184962 s, 56.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246857 s, 42.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00399978 s, 262 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104273 s, 101 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa0e708efa800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffffa0e708efa800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114028 s, 92.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134294 s, 78.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341541 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323536 s, 32.4 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg135-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg135-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (4s) == sanityn test 77c: check ORR NRS policy ================ 22:50:54 (1779072654) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.152765 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870525 s, 12.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852434 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913994 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109274 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109073 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110823 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112723 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150957 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145623 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145906 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156566 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156379 s, 6.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15417 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138668 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154993 s, 6.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa0e708efa800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffffa0e708efa800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176899 s, 5.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153305 s, 68.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00990038 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00437371 s, 240 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00343984 s, 305 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa0e708efa800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0036356 s, 288 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00482319 s, 217 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00332569 s, 315 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261665 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00423465 s, 248 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00428397 s, 245 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412827 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315047 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450952 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00671523 s, 156 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241313 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125138 s, 83.8 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg135-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg135-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.169217 s, 99.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445369 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599585 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647684 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957644 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900556 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116895 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940029 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111139 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110377 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117913 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127396 s, 8.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa0e708efa800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130074 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112972 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15894 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122196 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160072 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229854 s, 45.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242516 s, 43.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233191 s, 45.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312209 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301231 s, 34.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292655 s, 35.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300818 s, 34.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200692 s, 52.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029103 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270843 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117484 s, 89.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184722 s, 56.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00359061 s, 292 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00411995 s, 255 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321298 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130925 s, 80.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffffa0e708efa800.lock_unused_count=1 removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg135-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg135-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.151679 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337586 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121286 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127732 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146943 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154538 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160533 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14618 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165104 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156562 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174306 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153793 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163984 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174803 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152531 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175115 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153452 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320173 s, 32.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248577 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343618 s, 30.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198254 s, 52.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324804 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276977 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286724 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237779 s, 44.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254824 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045476 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427539 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288152 s, 36.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378813 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303581 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376122 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345522 s, 30.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa0e708efa800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffffa0e708efa800.lock_unused_count=1 removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg135-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg135-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (6s) == sanityn test 77d: check TRR nrs policy ================ 22:51:00 (1779072660) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.237139 s, 70.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445718 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901694 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107518 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108002 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104676 s, 10.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142761 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181329 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198696 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19045 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152493 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180955 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198766 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206883 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194475 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165146 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197715 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323469 s, 32.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262107 s, 40.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258928 s, 40.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244496 s, 42.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304618 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219128 s, 47.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00458166 s, 229 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352785 s, 29.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355525 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191595 s, 54.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317436 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201538 s, 52.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175851 s, 59.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205407 s, 51.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180872 s, 58.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00912396 s, 115 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa0e708efa800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffffa0e708efa800.lock_unused_count=1 removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg135-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg135-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.154349 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413144 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441415 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122787 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107365 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102955 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119038 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146557 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119606 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134142 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129194 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144348 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13889 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131557 s, 8.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa0e708efa800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffffa0e708efa800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144904 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159833 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156776 s, 6.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306011 s, 34.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182877 s, 57.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339965 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00991805 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031283 s, 33.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352427 s, 29.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215014 s, 48.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287103 s, 36.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317506 s, 33.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324453 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439569 s, 23.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367318 s, 28.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030561 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00870839 s, 120 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029814 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445644 s, 23.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa0e708efa800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffffa0e708efa800.lock_unused_count=1 removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg135-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg135-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.144002 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694183 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662428 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947343 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944247 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758717 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113767 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944587 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755137 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776445 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118624 s, 8.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffffa0e708efa800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145699 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113378 s, 9.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132253 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135539 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139829 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168556 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181008 s, 57.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164802 s, 63.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163955 s, 64.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229588 s, 45.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206977 s, 50.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168577 s, 62.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020576 s, 51.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173078 s, 60.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021899 s, 47.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238782 s, 43.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00483412 s, 217 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128828 s, 81.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00438165 s, 239 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030874 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249322 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299593 s, 35.0 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg135-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg135-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (5s) == sanityn test 77e: check TBF NID nrs policy ============ 22:51:05 (1779072665) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.201.35@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.19929 s, 84.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283337 s, 37.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031973 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112383 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147727 s, 7.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa0e708efa800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffffa0e708efa800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136058 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161444 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158658 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144833 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160276 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175293 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194492 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150435 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166282 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187129 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164873 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196553 s, 5.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa0e708efa800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffffa0e708efa800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166296 s, 63.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266749 s, 39.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120472 s, 87.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300579 s, 34.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191404 s, 54.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00391926 s, 268 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163089 s, 64.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229703 s, 45.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101311 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207817 s, 50.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132322 s, 79.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016529 s, 63.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00672697 s, 156 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0038824 s, 270 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00395409 s, 265 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363036 s, 28.9 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg135-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg135-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.166487 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761104 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923809 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823507 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104627 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126959 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142719 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145022 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163547 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151167 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164003 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17088 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176974 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172213 s, 6.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa0e708efa800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffffa0e708efa800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157512 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180496 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168692 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184369 s, 56.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172084 s, 60.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01718 s, 61.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162309 s, 64.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235431 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018158 s, 57.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210563 s, 49.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264011 s, 39.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215533 s, 48.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138695 s, 75.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101377 s, 103 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159109 s, 65.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00878346 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259826 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185016 s, 56.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116393 s, 90.1 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg135-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg135-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.147949 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381641 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711729 s, 14.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffffa0e708efa800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941862 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963673 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800251 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712037 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969897 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795729 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811017 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159449 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16517 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187447 s, 5.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164208 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161372 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00446438 s, 235 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00460831 s, 228 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00446302 s, 235 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257301 s, 40.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372419 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299768 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395816 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0038203 s, 274 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00780409 s, 134 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00411801 s, 255 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199368 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243528 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028528 s, 36.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa0e708efa800.lock_unused_count=10 ldlm.namespaces.lustre-OST0001-osc-ffffa0e708efa800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316797 s, 33.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00301778 s, 347 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413914 s, 25.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167912 s, 62.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202816 s, 5.2 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg135-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg135-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.181976 s, 92.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369588 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356887 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112491 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119879 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11251 s, 9.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa0e708efa800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffffa0e708efa800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13745 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123726 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112731 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153148 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130771 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146103 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136304 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146598 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134588 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128893 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141617 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00354178 s, 296 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00367925 s, 285 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00254072 s, 413 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288711 s, 36.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257112 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989048 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911198 s, 11.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338576 s, 31.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226673 s, 46.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208146 s, 50.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02198 s, 47.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014345 s, 73.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136908 s, 76.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110653 s, 94.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102546 s, 102 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00853573 s, 123 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa0e708efa800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffffa0e708efa800.lock_unused_count=2 removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg135-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg135-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (8s) == sanityn test 77f: check TBF JobID nrs policy ========== 22:51:13 (1779072673) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg135-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.151826 s, 111 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396776 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444032 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182845 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19045 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178568 s, 5.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19501 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202406 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211215 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233935 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00653493 s, 160 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111066 s, 94.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222855 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267324 s, 39.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142716 s, 73.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153181 s, 68.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233434 s, 4.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225376 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330611 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143885 s, 72.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022442 s, 46.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254834 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251051 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185239 s, 56.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0040009 s, 262 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255977 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104994 s, 99.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291493 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251578 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314594 s, 33.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164479 s, 63.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa0e708efa800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffffa0e708efa800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332178 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265458 s, 4.0 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg135-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg135-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg135-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.150745 s, 111 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063643 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118906 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15121 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138596 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177323 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133941 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14993 s, 7.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194265 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192717 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164695 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209887 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20804 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192625 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180434 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202147 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210929 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398026 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00234005 s, 448 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00246398 s, 426 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0030804 s, 340 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296968 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00319264 s, 328 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258169 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264754 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00493586 s, 212 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384787 s, 27.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216602 s, 48.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206305 s, 50.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00598541 s, 175 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019223 s, 54.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182129 s, 57.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148182 s, 70.8 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg135-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg135-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg135-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.140995 s, 119 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511926 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676803 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671517 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718043 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859536 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122255 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152117 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159445 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14706 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165485 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153665 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161021 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146206 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161988 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167379 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189652 s, 5.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281425 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245838 s, 42.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194617 s, 53.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291673 s, 36.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00448731 s, 234 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284137 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244623 s, 42.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314426 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315742 s, 33.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371165 s, 28.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280263 s, 37.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159069 s, 65.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa0e708efa800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00798689 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0044014 s, 238 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206738 s, 50.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.004175 s, 251 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg135-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg135-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg135-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.134854 s, 124 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341726 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358938 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323548 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687896 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732653 s, 14.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102058 s, 10.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887544 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962353 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896697 s, 11.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa0e708efa800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffffa0e708efa800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115069 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918975 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102434 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118194 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117436 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10701 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115847 s, 9.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0056318 s, 186 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193837 s, 54.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00409832 s, 256 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237626 s, 44.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018396 s, 57.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219263 s, 47.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322864 s, 32.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00779056 s, 135 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240148 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303092 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256964 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00690281 s, 152 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00325843 s, 322 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317802 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302887 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117113 s, 89.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffffa0e708efa800.lock_unused_count=1 removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg135-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg135-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (7s) == sanityn test 77g: Change TBF type directly ============ 22:51:20 (1779072680) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (3s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 22:51:23 (1779072683) oleg135-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg135-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg135-client: oleg135-server: ssh exited with exit code 2 oleg135-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg135-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg135-client: oleg135-server: ssh exited with exit code 22 oleg135-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg135-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg135-client: oleg135-server: ssh exited with exit code 22 oleg135-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg135-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg135-client: oleg135-server: ssh exited with exit code 22 PASS 77h (4s) == sanityn test 77i: Change rank of TBF rule ============= 22:51:27 (1779072687) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (7s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 22:51:34 (1779072694) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.144127 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256959 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334671 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376449 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814619 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686383 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806144 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072582 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805783 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775223 s, 13.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffffa0e708efa800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100325 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115396 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141492 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150928 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143532 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150341 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138957 s, 7.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00380862 s, 275 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00879257 s, 119 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00613649 s, 171 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216625 s, 48.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218399 s, 48.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309128 s, 33.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282337 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00609733 s, 172 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348671 s, 30.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036397 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00749531 s, 140 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236036 s, 44.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255663 s, 41.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125091 s, 83.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00985404 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017809 s, 58.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa0e708efa800.lock_unused_count=7 removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg135-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg135-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.89262 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4309 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (40s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 22:52:14 (1779072734) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg135-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg135-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg135-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg135-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg135-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.259108 s, 64.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345811 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907687 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107977 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122189 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117748 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113851 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134327 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130871 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121628 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126028 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11542 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122969 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12094 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126963 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135357 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162052 s, 6.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00516914 s, 203 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226846 s, 46.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258256 s, 40.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017882 s, 58.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170052 s, 61.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237027 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296528 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278435 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257816 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303556 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258841 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224615 s, 46.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293519 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230676 s, 45.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0036227 s, 289 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.003046 s, 344 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg135-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg135-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8252 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9618 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg135-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg135-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg135-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg135-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg135-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.256641 s, 65.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109527 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101252 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109792 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124195 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113545 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140635 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134008 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122885 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148897 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129484 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134708 s, 7.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117495 s, 8.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147335 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126581 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124895 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183299 s, 5.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01638 s, 64.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189444 s, 55.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01538 s, 68.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202658 s, 51.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224311 s, 46.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203219 s, 51.6 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178243 s, 58.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126456 s, 82.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102994 s, 102 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010681 s, 98.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247186 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00453019 s, 231 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344916 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136765 s, 76.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00737991 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00486246 s, 216 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg135-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg135-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.825 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9618 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (112s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 22:54:06 (1779072846) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg135-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg135-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg135-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg135-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg135-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.248042 s, 67.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471196 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409174 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624291 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642108 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924009 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109973 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090109 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992808 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106395 s, 9.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964667 s, 10.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10609 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102688 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102756 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116358 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100265 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13842 s, 7.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140244 s, 74.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00860978 s, 122 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151603 s, 69.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127006 s, 82.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013573 s, 77.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148162 s, 70.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197639 s, 53.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216337 s, 48.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152244 s, 68.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00591122 s, 177 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00360597 s, 291 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00796406 s, 132 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00679356 s, 154 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00607109 s, 173 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00278919 s, 376 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0043543 s, 241 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg135-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg135-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8239 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9601 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg135-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg135-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg135-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg135-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg135-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.255118 s, 65.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554943 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897727 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109664 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907899 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998852 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11123 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851883 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110669 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992973 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106546 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088371 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127833 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120762 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111361 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11825 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143321 s, 7.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018244 s, 57.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211106 s, 49.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194058 s, 54.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185922 s, 56.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168068 s, 62.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024438 s, 42.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163342 s, 64.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245803 s, 42.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184679 s, 56.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214531 s, 48.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103391 s, 101 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221569 s, 47.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281326 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00353994 s, 296 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00350198 s, 299 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109658 s, 95.6 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg135-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg135-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8232 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9642 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (113s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 22:55:59 (1779072959) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg135-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.168509 s, 99.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309125 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499408 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700796 s, 15.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffffa0e708efa800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105836 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827802 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109312 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126594 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14695 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142749 s, 7.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165594 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157457 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159616 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111375 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110224 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142408 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183375 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00463914 s, 226 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00279653 s, 375 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00343598 s, 305 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa0e708efa800.lock_unused_count=3 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401259 s, 26.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346942 s, 30.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025002 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00350802 s, 299 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032447 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470427 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021007 s, 49.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164767 s, 63.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229287 s, 45.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130128 s, 80.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010166 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109718 s, 95.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00395168 s, 265 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg135-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg135-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.90449 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73099 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.201.35@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.201.35@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.158572 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372562 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574633 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571318 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784905 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093907 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760078 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881513 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103023 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766894 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112091 s, 9.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa0e708efa800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffffa0e708efa800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134172 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131959 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130109 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145523 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152466 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148819 s, 7.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102252 s, 10.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934785 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108005 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963817 s, 10.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985917 s, 10.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107602 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967648 s, 10.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103708 s, 10.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103119 s, 10.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102189 s, 10.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994444 s, 10.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103004 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107542 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109565 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104153 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107057 s, 9.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa0e708efa800.lock_unused_count=7 removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg135-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg135-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.89812 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73296 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.201.35@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg135-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.159457 s, 105 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526067 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102963 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114249 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119995 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13152 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123681 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136824 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111364 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129797 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140494 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137165 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13238 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129212 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146863 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142433 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139975 s, 7.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0000-osc-ffffa0e708efa800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136712 s, 76.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126549 s, 82.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160173 s, 65.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164833 s, 63.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00493331 s, 213 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205906 s, 50.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00979086 s, 107 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104513 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216701 s, 48.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00978481 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157565 s, 66.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00826752 s, 127 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00921159 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00298664 s, 351 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016658 s, 62.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00930168 s, 113 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg135-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg135-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.90278 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.88377 s, 21.5 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg135-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.163249 s, 103 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817211 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712975 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665665 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842593 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109317 s, 9.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109566 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996852 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138641 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141498 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154882 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145792 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151982 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149052 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162719 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147267 s, 7.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa0e708efa800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffffa0e708efa800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17158 s, 6.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204305 s, 51.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134797 s, 77.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151723 s, 69.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158423 s, 66.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121535 s, 86.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179931 s, 58.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254431 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206468 s, 50.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108765 s, 96.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198294 s, 52.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00403762 s, 260 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243558 s, 43.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00910267 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00625183 s, 168 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00384505 s, 273 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00565229 s, 186 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa0e708efa800.lock_unused_count=1 removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg135-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg135-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.75702 s, 10.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73614 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.89032 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73439 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg135-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.156736 s, 107 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497845 s, 21.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105952 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12412 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107154 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132711 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120523 s, 8.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128489 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144233 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12487 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138513 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123411 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130315 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148312 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132565 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141758 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138959 s, 7.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149354 s, 70.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210007 s, 49.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186883 s, 56.1 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210491 s, 49.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227832 s, 46.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202704 s, 51.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021234 s, 49.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296692 s, 35.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221803 s, 47.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00510907 s, 205 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00739878 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0093354 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156261 s, 67.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150352 s, 69.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00536348 s, 196 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0058795 s, 178 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg135-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg135-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6571 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.436 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg135-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.154136 s, 109 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429808 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619498 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073947 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068737 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756647 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859033 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849116 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837311 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596143 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076526 s, 13.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffffa0e708efa800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120796 s, 8.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118169 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154744 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151874 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150423 s, 7.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104494 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206092 s, 50.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147629 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00868048 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160006 s, 65.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259959 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214557 s, 48.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235676 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244182 s, 42.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00732632 s, 143 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354327 s, 29.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298823 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00569122 s, 184 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307843 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00327392 s, 320 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180529 s, 58.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160186 s, 65.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg135-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg135-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.90173 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73789 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg135-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.14875 s, 113 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146664 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224394 s, 4.7 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221098 s, 4.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224784 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216787 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215282 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213837 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210875 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195417 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228986 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210465 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238046 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221256 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247802 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223073 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223916 s, 4.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343726 s, 30.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039496 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354334 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340124 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350112 s, 29.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504584 s, 20.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554437 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451742 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430641 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042592 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483965 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474835 s, 22.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496924 s, 21.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474303 s, 22.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffffa0e708efa800.lock_unused_count=1 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499315 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501979 s, 20.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg135-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg135-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7658 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.75195 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74885 s, 10.8 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74704 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (273s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 23:00:32 (1779073232) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg135-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.144383 s, 116 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839017 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106334 s, 9.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10355 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987126 s, 10.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114126 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120115 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972937 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118109 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13751 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126017 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105825 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110069 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136781 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121993 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12725 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136742 s, 7.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0000-osc-ffffa0e708efa800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffffa0e708efa800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150177 s, 69.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179951 s, 58.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287265 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197125 s, 53.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147493 s, 71.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206899 s, 50.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207005 s, 50.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209642 s, 50.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265479 s, 39.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00691623 s, 152 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277394 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030098 s, 34.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266858 s, 39.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280307 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250833 s, 41.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271469 s, 38.6 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg135-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg135-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.90643 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74956 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn Waiting for MDT destroys to complete PASS 77kb (26s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 23:00:58 (1779073258) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.201.35@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.201.35@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.140292 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379493 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607575 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056057 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076679 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099304 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989787 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125575 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104245 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111425 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145691 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138762 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124247 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15738 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1528 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147878 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142038 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051986 s, 20.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599169 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570961 s, 18.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509449 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553083 s, 19.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557157 s, 18.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601921 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619614 s, 16.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063325 s, 16.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058224 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608529 s, 17.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588011 s, 17.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059721 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657965 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622658 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626082 s, 16.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa0e708efa800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffffa0e708efa800.lock_unused_count=2 removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg135-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg135-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.90212 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73263 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn Waiting for MDT destroys to complete PASS 77kc (26s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 23:01:24 (1779073284) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.201.35@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg135-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.165375 s, 101 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046964 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530944 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794805 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783325 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869037 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924285 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863957 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802017 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099498 s, 10.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12586 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161008 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167132 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161724 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174917 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00911833 s, 115 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156826 s, 66.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104784 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192447 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232617 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101153 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252462 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108468 s, 96.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262075 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274753 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315493 s, 33.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00502004 s, 209 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252107 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309976 s, 33.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa0e708efa800.lock_unused_count=8 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296678 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205509 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312931 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038936 s, 26.9 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg135-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg135-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.90162 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97478 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn Waiting for MDT destroys to complete PASS 77kd (25s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 23:01:50 (1779073310) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg135-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.154404 s, 109 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830512 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144661 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195626 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205888 s, 5.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187417 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189554 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212918 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219059 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24203 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203608 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200108 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220433 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232231 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25509 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236333 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252899 s, 4.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0000-osc-ffffa0e708efa800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffffa0e708efa800.lock_unused_count=5 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361328 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336146 s, 31.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353837 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435281 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417132 s, 25.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321072 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374358 s, 28.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039209 s, 26.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327339 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198735 s, 52.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403077 s, 26.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222819 s, 47.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169709 s, 61.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157874 s, 66.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396908 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260594 s, 40.2 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg135-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg135-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74962 s, 10.8 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73396 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.89016 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.72811 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn Waiting for MDT destroys to complete PASS 77ke (57s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 23:02:46 (1779073366) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg135-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.147671 s, 114 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284587 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617763 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813408 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887853 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910079 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716945 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867409 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708281 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860697 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129543 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131563 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142117 s, 7.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130255 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149884 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13944 s, 7.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181888 s, 57.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143626 s, 73.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213348 s, 49.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196851 s, 53.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195193 s, 53.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187322 s, 56.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017677 s, 59.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178094 s, 58.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00665374 s, 158 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164881 s, 63.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198741 s, 52.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245797 s, 42.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260778 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022853 s, 45.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00359558 s, 292 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00416016 s, 252 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg135-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg135-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6511 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9554 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn Waiting for MDT destroys to complete PASS 77kf (51s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 23:03:37 (1779073417) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg135-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.1744 s, 96.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557618 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840317 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870771 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846051 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719564 s, 14.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732457 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783071 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10189 s, 10.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701338 s, 15.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115277 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137834 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00891143 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016807 s, 62.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187228 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00900751 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124851 s, 84.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180247 s, 58.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159928 s, 65.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193095 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178192 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00957185 s, 110 MB/s ldlm.namespaces.lustre-OST0001-osc-ffffa0e708efa800.lock_unused_count=8 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185651 s, 5.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00824431 s, 127 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162063 s, 64.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128391 s, 81.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0061227 s, 171 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129033 s, 81.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137441 s, 76.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015318 s, 68.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133402 s, 78.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152832 s, 68.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201635 s, 5.2 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg135-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg135-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.90634 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73842 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg135-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.177819 s, 94.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538246 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468836 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148303 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168681 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204576 s, 5.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa0e708efa800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffffa0e708efa800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204562 s, 5.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215929 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217259 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224887 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259207 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246428 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25513 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2396 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250837 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25464 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281092 s, 37.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283913 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345073 s, 30.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308258 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320156 s, 32.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375427 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382787 s, 27.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387488 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355179 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497876 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422151 s, 24.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037188 s, 28.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421393 s, 24.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437662 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042009 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447978 s, 23.4 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg135-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg135-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76494 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74348 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7428 s, 10.8 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73524 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn Waiting for MDT destroys to complete PASS 77kg (87s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 23:05:04 (1779073504) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (4s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 23:05:08 (1779073508) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg135-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.77005 s, 278 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg135-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0089 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (51s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 23:05:59 (1779073559) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.151157 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553572 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124086 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127986 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901636 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125893 s, 8.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa0e708efa800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216989 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194115 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213252 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211111 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243505 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238661 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289649 s, 36.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251816 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039881 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037772 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235165 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266014 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04202 s, 25.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477553 s, 22.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039399 s, 26.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416888 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248492 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430947 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331018 s, 31.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043726 s, 24.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055801 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23413 s, 4.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266538 s, 3.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254734 s, 41.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406668 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00836632 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00382537 s, 274 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg135-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg135-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.91254 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97083 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.150612 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414656 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414027 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823045 s, 12.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffffa0e708efa800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffffa0e708efa800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114424 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127716 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1161 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113882 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122496 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157698 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155268 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154419 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137808 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167229 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164802 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150558 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166986 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020293 s, 51.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00486736 s, 215 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244822 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149931 s, 69.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282311 s, 37.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190626 s, 55.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297635 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0034113 s, 307 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00451297 s, 232 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103954 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00547941 s, 191 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00323987 s, 324 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384096 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00952427 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00310729 s, 337 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00924393 s, 113 MB/s ldlm.namespaces.lustre-OST0001-osc-ffffa0e708efa800.lock_unused_count=4 removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg135-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg135-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.90378 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97883 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (50s) == sanityn test 77o: Changing rank should not panic ====== 23:06:49 (1779073609) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (4s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 23:06:53 (1779073613) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (41s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 23:07:34 (1779073654) mds.MDS.mdt.nrs_policies=tbf oleg135-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg135-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg135-client: oleg135-server: ssh exited with exit code 36 oleg135-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg135-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg135-client: oleg135-server: ssh exited with exit code 22 oleg135-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg135-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg135-client: oleg135-server: ssh exited with exit code 22 oleg135-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg135-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg135-client: oleg135-server: ssh exited with exit code 22 oleg135-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg135-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg135-client: oleg135-server: ssh exited with exit code 22 oleg135-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg135-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg135-client: oleg135-server: ssh exited with exit code 22 oleg135-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg135-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg135-client: oleg135-server: ssh exited with exit code 22 oleg135-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg135-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg135-client: oleg135-server: ssh exited with exit code 22 oleg135-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg135-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg135-client: oleg135-server: ssh exited with exit code 22 oleg135-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg135-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg135-client: oleg135-server: ssh exited with exit code 22 oleg135-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg135-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg135-client: oleg135-server: ssh exited with exit code 22 oleg135-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg135-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg135-client: oleg135-server: ssh exited with exit code 22 oleg135-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg135-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg135-client: oleg135-server: ssh exited with exit code 22 oleg135-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg135-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg135-client: oleg135-server: ssh exited with exit code 22 oleg135-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg135-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg135-client: oleg135-server: ssh exited with exit code 22 oleg135-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg135-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg135-client: oleg135-server: ssh exited with exit code 22 oleg135-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg135-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg135-client: oleg135-server: ssh exited with exit code 22 oleg135-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg135-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg135-client: oleg135-server: ssh exited with exit code 22 oleg135-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg135-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg135-client: oleg135-server: ssh exited with exit code 22 oleg135-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg135-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg135-client: oleg135-server: ssh exited with exit code 22 oleg135-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg135-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg135-client: oleg135-server: ssh exited with exit code 22 oleg135-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg135-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg135-client: oleg135-server: ssh exited with exit code 22 oleg135-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg135-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg135-client: oleg135-server: ssh exited with exit code 22 oleg135-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg135-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg135-client: oleg135-server: ssh exited with exit code 22 oleg135-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg135-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg135-client: oleg135-server: ssh exited with exit code 22 oleg135-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg135-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg135-client: oleg135-server: ssh exited with exit code 22 oleg135-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg135-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg135-client: oleg135-server: ssh exited with exit code 22 oleg135-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg135-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg135-client: oleg135-server: ssh exited with exit code 22 oleg135-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg135-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg135-client: oleg135-server: ssh exited with exit code 22 oleg135-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg135-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg135-client: oleg135-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (12s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 23:07:46 (1779073666) striped dir -i0 -c1 -H all_char /mnt/lustre/d77r.sanityn mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 339734 kill md_thread xargs PASS 77r (40s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 23:08:26 (1779073706) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (2s) == sanityn test 79: xattr: intent error ================== 23:08:28 (1779073708) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d79.sanityn File: /mnt/lustre/d79.sanityn Size: 8192 Blocks: 16 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798763028 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-17 23:08:29.000000000 -0400 Modify: 2026-05-17 23:08:29.000000000 -0400 Change: 2026-05-17 23:08:29.000000000 -0400 Birth: 2026-05-17 23:08:29.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (13s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 23:08:41 (1779073721) total: 10 open/close in 0.03 seconds: 294.67 ops/second multiop /mnt/lustre2/d80a.sanityn/f80a.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8189 Finish migration, then checking.. PASS 80a (4s) == sanityn test 80b: Accessing directory during migration ========================================================== 23:08:45 (1779073725) start migration thread 347478 accessing the migrating directory for 1 minute... concurrent access 2047 failures, 2651 successes /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 347478 Killed ( while true; do mdt_idx=$((RANDOM % MDSCOUNT)); $LFS migrate -m $mdt_idx $migdir1 &> /dev/null; done ) (wd: ~) PASS 80b (62s) == sanityn test 81a: rename and stat under striped directory ========================================================== 23:09:47 (1779073787) stat: cannot statx 'd0/0': No such file or directory File: d0/0 Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205289285085 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-17 23:09:48.000000000 -0400 Modify: 2026-05-17 23:09:48.000000000 -0400 Change: 2026-05-17 23:09:48.000000000 -0400 Birth: 2026-05-17 23:09:48.000000000 -0400 PASS 81a (3s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 23:09:50 (1779073790) total: 2500 open/close in 5.32 seconds: 469.63 ops/second /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 354190 Killed ( while true; do touch $DIR1/$tdir; done ) (wd: ~) PASS 81b (46s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 23:10:37 (1779073837) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (2s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 23:10:38 (1779073838) total: 2500 open/close in 8.61 seconds: 290.27 ops/second total: 2500 open/close in 8.12 seconds: 307.96 ops/second snapshot_time 1779073885.099061293 secs.nsecs start_time 1779073839.060814608 secs.nsecs elapsed_time 46.038246685 secs.nsecs open 2505 samples [usecs] 75 2039 973390 420470386 close 2504 samples [usecs] 51 3377 272576 43760720 mknod 2501 samples [usecs] 151 1952 814276 296377780 unlink 2500 samples [usecs] 220 13447 1168696 1511753128 mkdir 2 samples [usecs] 926 4080 5006 17503876 rmdir 2 samples [usecs] 1737 1844 3581 6417505 rename 2500 samples [usecs] 211 7928 2979858 5625633790 getattr 3764 samples [usecs] 0 26749 129588 728901804 statfs 9 samples [usecs] 10 30 169 3531 parallel_rename_file 2500 samples [usecs] 211 7928 2979858 5625633790 crossdir_rename 2500 samples [usecs] 211 7928 2979858 5625633790 snapshot_time 1779073885.099167406 secs.nsecs start_time 1779073839.060939728 secs.nsecs elapsed_time 46.038227678 secs.nsecs open 2504 samples [usecs] 49 4362 957815 429626927 close 2504 samples [usecs] 50 432 271317 33982281 mknod 2500 samples [usecs] 149 4133 801851 306018841 unlink 2500 samples [usecs] 220 24699 1164910 1870417902 mkdir 2 samples [usecs] 1021 8897 9918 80199050 rmdir 2 samples [usecs] 1932 83218 85150 6928968148 rename 2500 samples [usecs] 212 5349 2801481 4619172377 getattr 3757 samples [usecs] 8 1705 93852 7568554 statfs 9 samples [usecs] 13 37 173 3783 parallel_rename_file 2500 samples [usecs] 212 5349 2801481 4619172377 crossdir_rename 2500 samples [usecs] 212 5349 2801481 4619172377 PASS 81d (49s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 23:11:27 (1779073887) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.8189 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.8189 PASS 82 (3s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 23:11:30 (1779073890) start pid 368660 to create/unlink striped directory start pid 368661 to stat striped directory /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5797: 368660 Terminated ( cd $DIR1; while true; do $LFS mkdir -i1 -c2 $tdir; rmdir $tdir; done ) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5797: 368661 Terminated ( cd $DIR2; while true; do stat $tdir > /dev/null 2>&1; done ) PASS 83 (123s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 23:13:33 (1779074013) fail_loc=0x8000060b PASS 84 (7s) == sanityn test 85: Lustre API root cache race =========== 23:13:40 (1779074020) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1779074021 Finishing test test1 at 1779074021 PASS 85 (3s) == sanityn test 90: open/create and unlink striped directory ========================================================== 23:13:43 (1779074023) start pid 379421 to open/create under striped directory start pid 379423 to unlink striped directory /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5864: 379421 Terminated ( cd $DIR1; while true; do $LFS mkdir -c$MDSCOUNT $tdir > /dev/null 2>&1; touch $tdir/f{0..3} > /dev/null 2>&1; done ) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5864: 379423 Terminated ( cd $DIR2; while true; do rm -rf $tdir > /dev/null 2>&1; done ) PASS 90 (183s) == sanityn test 91: chmod and unlink striped directory === 23:16:46 (1779074206) start pid 389507 to chmod striped directory start pid 389508 to unlink striped directory /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5901: 389507 Terminated ( cd $DIR1; while true; do $LFS mkdir -c$MDSCOUNT $tdir > /dev/null 2>&1; chmod go+w $tdir > /dev/null 2>&1; done ) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5901: 389508 Terminated ( cd $DIR2; while true; do rm -rf $tdir > /dev/null 2>&1; done ) PASS 91 (183s) == sanityn test 92: create remote directory under orphan directory ========================================================== 23:19:49 (1779074389) fail_loc=0x1408 mkdir: cannot create directory '/mnt/lustre2/d92.sanityn/dir': No such file or directory lfs setdirstripe: cannot resolve path '/mnt/lustre2/d92.sanityn/remote_dir': No such file or directory (2) lfs setdirstripe: '/mnt/lustre2/d92.sanityn/remote_dir' is not on a Lustre filesystem: No such file or directory (2) lfs setdirstripe: cannot create dir '/mnt/lustre2/d92.sanityn/remote_dir': No such file or directory fail_loc=0 PASS 92 (3s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 23:19:52 (1779074392) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5169 0x1431 0x2c0000400 0 5186 0x1442 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6123 0x17eb 0x2c0000401 0 6134 0x17f6 0x280000401 rc1=0 and rc2=0 PASS 93 (8s) == sanityn test 94: signal vs CP callback race =========== 23:20:01 (1779074401) osc.lustre-OST0000-osc-ffffa0e708efa800.idle_timeout=0 osc.lustre-OST0000-osc-ffffa0e71017f800.idle_timeout=0 osc.lustre-OST0001-osc-ffffa0e708efa800.idle_timeout=0 osc.lustre-OST0001-osc-ffffa0e71017f800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00834479 s, 982 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.09208 s, 1.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5988: 403861 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00442072 s, 927 kB/s osc.lustre-OST0000-osc-ffffa0e708efa800.idle_timeout=debug osc.lustre-OST0000-osc-ffffa0e71017f800.idle_timeout=debug osc.lustre-OST0001-osc-ffffa0e708efa800.idle_timeout=debug osc.lustre-OST0001-osc-ffffa0e71017f800.idle_timeout=debug PASS 94 (10s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 23:20:10 (1779074410) llite.lustre-ffffa0e708efa800.fast_read=0 llite.lustre-ffffa0e71017f800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0164339 s, 128 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffffa0e708efa800.fast_read=1 llite.lustre-ffffa0e71017f800.fast_read=1 PASS 95a (15s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 23:20:25 (1779074425) llite.lustre-ffffa0e708efa800.fast_read=0 llite.lustre-ffffa0e71017f800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00152461 s, 8.1 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffffa0e708efa800.fast_read=1 llite.lustre-ffffa0e71017f800.fast_read=1 PASS 95b (8s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 23:20:33 (1779074433) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (1s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 23:20:34 (1779074434) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0020752 s, 2.0 MB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (3s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 23:20:37 (1779074437) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.031076 s, 67.5 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (3s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 23:20:40 (1779074440) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0317998 s, 65.9 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (2s) == sanityn test 100e: DoM: read on open and file size ==== 23:20:42 (1779074442) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00233527 s, 5.1 MB/s PASS 100e (3s) == sanityn test 101a: Discard DoM data on unlink ========= 23:20:45 (1779074445) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000605893 s, 6.8 MB/s PASS 101a (3s) == sanityn test 101b: Discard DoM data on rename ========= 23:20:48 (1779074448) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000649144 s, 6.3 MB/s PASS 101b (2s) == sanityn test 101c: Discard DoM data on close-unlink === 23:20:50 (1779074450) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000463721 s, 8.8 MB/s PASS 101c (4s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 23:20:54 (1779074454) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: 02 04 00 00 02 00 00 00 21 32 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000402:0x3221:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102.sanityn.remote fh_bytes: 32 fh_type: 151 fh_data: 02 04 00 40 02 00 00 00 78 35 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x240000402:0x3578:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102.sanityn.remote/subdir fh_bytes: 32 fh_type: 151 fh_data: 02 04 00 00 02 00 00 00 22 32 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000402:0x3222:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102.sanityn.stripe fh_bytes: 32 fh_type: 151 fh_data: 02 04 00 40 02 00 00 00 79 35 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x240000402:0x3579:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102.sanityn.stripe/subdir fh_bytes: 32 fh_type: 151 fh_data: 02 04 00 40 02 00 00 00 7a 35 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x240000402:0x357a:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! PASS 102 (3s) == sanityn test 103: Test size correctness with lockahead ========================================================== 23:20:58 (1779074458) striped dir -i1 -c2 -H crush2 /mnt/lustre/d103.sanityn fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1779074459 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1779074461 Finishing test test23 at 1779074461 PASS 103 (8s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 23:21:06 (1779074466) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00447083 s, 229 kB/s oleg135-server: debugfs 1.47.3-wc2 (11-Nov-2025) STAT a|m|ctime 1779074467 1779074467 1779074467 MDT a|m|ctime 1779074467 1779074467 1779074467 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00274637 s, 373 kB/s oleg135-server: debugfs 1.47.3-wc2 (11-Nov-2025) STAT a|m|ctime 1779074467 1779074470 1779074470 MDT a|m|ctime 1779074467 1779074470 1779074470 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0070436 s, 145 kB/s oleg135-server: debugfs 1.47.3-wc2 (11-Nov-2025) STAT a|m|ctime 1779074467 1779074474 1779074474 MDT a|m|ctime 1779074467 1779074474 1779074474 oleg135-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg135-server: debugfs 1.47.3-wc2 (11-Nov-2025) STAT a|m|ctime 1779074467 1779074474 1779074474 MDT a|m|ctime 1779074467 1779074474 1779074474 oleg135-server: debugfs 1.47.3-wc2 (11-Nov-2025) STAT a|m|ctime 1779074467 1779074474 1779074483 MDT a|m|ctime 1779074467 1779074474 1779074483 PASS 104 (19s) == sanityn test 105: Glimpse and lock cancel race ======== 23:21:25 (1779074485) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d105.sanityn fail_loc=0x416 PASS 105 (23s) == sanityn test 106a: Verify the btime via statx() ======= 23:21:48 (1779074508) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00404784 s, 253 kB/s oleg135-server: debugfs 1.47.3-wc2 (11-Nov-2025) Inode: 5235 Type: regular Mode: 0644 Flags: 0x80000 Generation: 4209450167 Version: 0x00000002:00017fd0 User: 0 Group: 0 Project: 0 Size: 0 File ACL: 0 Links: 1 Blockcount: 0 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x6a0a85cd:00000000 -- Sun May 17 23:21:49 2026 atime: 0x6a0a85cd:00000000 -- Sun May 17 23:21:49 2026 mtime: 0x6a0a85cd:00000000 -- Sun May 17 23:21:49 2026 crtime: 0x6a0a85cc:d4e86500 -- Sun May 17 23:21:48 2026 Size of extra inode fields: 32 Extended attributes: lma: fid=[0x200000402:0x3226:0x0] compat=0 incompat=0 trusted.lov (56) = d0 0b d1 0b 01 00 00 00 26 32 00 00 00 00 00 00 02 04 00 00 02 00 00 00 00 00 40 00 01 00 00 00 01 04 00 c0 02 00 00 00 f1 17 00 00 00 00 00 00 00 00 00 00 01 00 00 00 user.job (4) = "dd.0" linkea: idx=0 parent=[0x200000007:0x1:0x0] name='f106a.sanityn' trusted.som (24) = 04 00 00 00 00 00 00 00 00 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 EXTENTS: PASS 106a (3s) == sanityn test 106b: Glimpse RPCs test for statx ======== 23:21:51 (1779074511) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00627165 s, 163 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 8 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205289292327 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-17 23:21:51.000000000 -0400 Modify: 2026-05-17 23:21:51.000000000 -0400 Change: 2026-05-17 23:21:51.000000000 -0400 Birth: 2026-05-17 23:21:51.000000000 -0400 /mnt/lustre/f106b.sanityn 144115205289292327 -rw-r--r-- 1 0 0 1779074511 1779074511 1779074511 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 8 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205289292327 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-17 23:21:51.000000000 -0400 Modify: 2026-05-17 23:21:51.000000000 -0400 Change: 2026-05-17 23:21:51.000000000 -0400 Birth: 2026-05-17 23:21:51.000000000 -0400 1779074511 1024 8 PASS 106b (3s) == sanityn test 106c: Verify statx attributes mask ======= 23:21:54 (1779074514) PASS 106c (3s) == sanityn test 107a: Basic grouplock conflict =========== 23:21:57 (1779074517) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.127806 s, 82.0 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6141 0x17fd 0x280000401 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.8189 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.8189 First grouplock blocks second one PASS 107a (5s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 23:22:02 (1779074522) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.125033 s, 83.9 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6131 0x17f3 0x2c0000401 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.8189 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (9s) == sanityn test 108a: lseek: parallel updates ============ 23:22:11 (1779074531) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00466984 s, 2.1 MB/s llite.lustre-ffffa0e708efa800.hybrid_io=0 llite.lustre-ffffa0e71017f800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.22377 s, 2.0 MB/s PASS 108a (7s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 23:22:18 (1779074538) 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre (opts:) 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) Starting race between client mount instances (50 iterations): Iteration 1 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre (opts:) Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 2 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre (opts:) Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 3 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre (opts:) Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 4 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre (opts:) Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 5 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre (opts:) Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 6 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre (opts:) Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 7 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg135-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 8 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg135-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 9 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg135-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 10 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg135-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 11 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg135-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 12 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre (opts:) Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 13 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg135-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 14 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg135-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 15 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg135-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 16 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre (opts:) Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 17 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg135-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 18 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg135-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 19 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg135-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 20 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre (opts:) Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 21 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre (opts:) Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 22 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre (opts:) Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 23 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg135-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 24 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg135-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 25 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre (opts:) Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 26 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre (opts:) Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 27 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg135-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 28 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre (opts:) Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 29 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg135-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 30 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre (opts:) Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 31 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg135-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 32 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre (opts:) Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 33 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre (opts:) Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 34 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre (opts:) Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 35 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg135-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 36 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre (opts:) Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 37 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre (opts:) Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 38 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg135-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 39 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg135-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 40 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre (opts:) Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 41 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre (opts:) Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 42 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg135-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 43 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre (opts:) Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 44 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre (opts:) Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 45 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre (opts:) Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 46 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg135-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 47 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg135-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 48 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre (opts:) Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 49 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre (opts:) Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Iteration 50 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 192.168.201.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre (opts:) Stopping client oleg135-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 PASS 109 (505s) == sanityn test 110: do not grant another lock on resend ========================================================== 23:30:43 (1779075043) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934275 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-17 23:30:44.000000000 -0400 Modify: 2026-05-17 23:30:44.000000000 -0400 Change: 2026-05-17 23:30:44.000000000 -0400 Birth: 2026-05-17 23:30:43.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg135-client.virtnet oleg135-client.virtnet PASS 110 (66s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 23:31:49 (1779075109) fail_loc=0x8000018a /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6784: 476584 Terminated $MULTIOP $DIR2/$tdir/mdt0dir/foodir/file2 Ow4096_c PASS 111 (7s) == sanityn test 112: update max-inherit in default LMV === 23:31:56 (1779075116) lmv.lustre-clilmv-ffffa0e71017b800.qos_threshold_rr=90 lmv.lustre-clilmv-ffffa0e72040d800.qos_threshold_rr=90 PASS 112 (4s) == sanityn test 113: check servers of specified fs ======= 23:32:01 (1779075121) lustre-OST0000-osc-ffffa0e72040d800 active. lustre-OST0001-osc-ffffa0e72040d800 active. lustre-OST0000-osc-ffffa0e72040d800 active. lustre-OST0001-osc-ffffa0e72040d800 active. MGC192.168.201.135@tcp active. PASS 113 (3s) == sanityn test 114: implicit default LMV inherit ======== 23:32:03 (1779075123) /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1 lmv_stripe_count: 2 lmv_stripe_offset: 2 lmv_hash_type: none lmv_max_inherit: 2 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1 lmv_stripe_count: 2 lmv_stripe_offset: 2 lmv_hash_type: none lmv_max_inherit: 2 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 PASS 114 (11s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 23:32:14 (1779075134) total: 3000 create in 6.14 seconds: 488.54 ops/second oleg135-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg135-server: htree_dump_int_block: Ext2 directory block not found while calculating checksum for htree root oleg135-server: USE: f566 f1523 fail_loc=0x80002401 fail_val=5 mkdir: cannot create directory '/mnt/lustre/d115.sanityn/f1523': File exists PASS 115 (17s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 23:32:31 (1779075151) llite.lustre-ffffa0e71017b800.intent_mkdir=1 llite.lustre-ffffa0e72040d800.intent_mkdir=1 MD layout /mnt/lustre/d116.sanityn: lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x200000bd0:0x638:0x0] 1 [0x240000404:0x5ae:0x0] mkdir /mnt/lustre/d116.sanityn/tdir0 setdirstripe -D -i1 /mnt/lustre2/d116.sanityn/tdir0 mkdir /mnt/lustre/d116.sanityn/tdir0/tdir11 lmv_stripe_count: 0 lmv_stripe_offset: 0 lmv_hash_type: none mdtidx FID[seq:oid:ver] lmv_stripe_count: 0 lmv_stripe_offset: 1 lmv_hash_type: none mdtidx FID[seq:oid:ver] PASS 116 (3s) == sanityn test 117a: TCU: Init and enable Trash Can on MDTs ========================================================== 23:32:34 (1779075154) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 PASS 117a (4s) == sanityn test 117b: Move regular file and empty dir into trash can dir ========================================================== 23:32:38 (1779075158) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain llite.lustre-ffffa0e71017b800.trash_file_read=1 llite.lustre-ffffa0e72040d800.trash_file_read=1 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0927911 s, 45.2 MB/s /mnt/lustre/.lustre/.ltrash: MDT0000 MDT0001 /mnt/lustre/.lustre/.ltrash/MDT0000: 0x200000bd1:0xbcc:0x0 /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbcc:0x0: tdir1 tdir2 tf1 tf2 tf3 /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbcc:0x0/tdir1: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbcc:0x0/tdir2: /mnt/lustre/.lustre/.ltrash/MDT0001: llite.lustre-ffffa0e71017b800.trash_file_read=0 llite.lustre-ffffa0e72040d800.trash_file_read=0 mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117b (7s) == sanityn test 117c: Move deleted tree with multiple levels into trash ========================================================== 23:32:46 (1779075166) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain /mnt/lustre/.lustre/.ltrash/MDT0000: 0x200000bd1:0xbcc:0x0 0x200000bd1:0xbd2:0x0 /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbcc:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbd2:0x0: tdir1 tdir2 /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbd2:0x0/tdir1: tdir11 tdir12 tfile13 /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbd2:0x0/tdir1/tdir11: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbd2:0x0/tdir1/tdir12: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbd2:0x0/tdir2: mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117c (7s) == sanityn test 117d: Per-User Trash can Type testing ==== 23:32:52 (1779075172) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid Trash Can Type: uid running as uid/gid/euid/egid 500/500/500/500, groups: 500 [mkdir] [/mnt/lustre/d117d.sanityn/tdir2] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [mkdir] [/mnt/lustre/d117d.sanityn/tdir2/tdir21] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [mkdir] [/mnt/lustre/d117d.sanityn/tdir2/tdir22] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d117d.sanityn/tdir2/tfile23] uid1: 0 uid2: 500 /mnt/lustre/.lustre/.ltrash/MDT0000: 0 0x200000bd1:0xbcc:0x0 0x200000bd1:0xbd2:0x0 500 /mnt/lustre/.lustre/.ltrash/MDT0000/0: 0x200000bd1:0xbdb:0x0 /mnt/lustre/.lustre/.ltrash/MDT0000/0/0x200000bd1:0xbdb:0x0: tdir11 tdir12 tfile13 /mnt/lustre/.lustre/.ltrash/MDT0000/0/0x200000bd1:0xbdb:0x0/tdir11: /mnt/lustre/.lustre/.ltrash/MDT0000/0/0x200000bd1:0xbdb:0x0/tdir12: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbcc:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbd2:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/500: 0x200000bd1:0xbdf:0x0 /mnt/lustre/.lustre/.ltrash/MDT0000/500/0x200000bd1:0xbdf:0x0: tdir21 tdir22 tfile23 /mnt/lustre/.lustre/.ltrash/MDT0000/500/0x200000bd1:0xbdf:0x0/tdir21: /mnt/lustre/.lustre/.ltrash/MDT0000/500/0x200000bd1:0xbdf:0x0/tdir22: mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117d (7s) == sanityn test 117e: Undeleted dir in trash should keep its original xattrs ========================================================== 23:32:59 (1779075179) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain /mnt/lustre/.lustre/.ltrash/MDT0000: 0 0x200000bd1:0xbcc:0x0 0x200000bd1:0xbd2:0x0 0x200000bd1:0xbe3:0x0 500 /mnt/lustre/.lustre/.ltrash/MDT0000/0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbcc:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbd2:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe3:0x0: tdir1 /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe3:0x0/tdir1: tdir11 tfile12 /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe3:0x0/tdir1/tdir11: /mnt/lustre/.lustre/.ltrash/MDT0000/500: trusted.name11="value11" trusted.name12="value12" trusted.name1="value1" mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117e (7s) == sanityn test 117f: Uncache the dentry under the trash dir ========================================================== 23:33:07 (1779075187) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain /mnt/lustre/.lustre/.ltrash/MDT0000: 0 0x200000bd1:0xbcc:0x0 0x200000bd1:0xbd2:0x0 0x200000bd1:0xbe3:0x0 0x200000bd1:0xbe7:0x0 500 /mnt/lustre/.lustre/.ltrash/MDT0000/0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbcc:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbd2:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0: tdir1 tfile3 /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0/tdir1: /mnt/lustre/.lustre/.ltrash/MDT0000/500: tdir1 tfile3 /mnt/lustre/.lustre/.ltrash/MDT0000: 0 0x200000bd1:0xbcc:0x0 0x200000bd1:0xbd2:0x0 0x200000bd1:0xbe3:0x0 0x200000bd1:0xbe7:0x0 500 /mnt/lustre/.lustre/.ltrash/MDT0000/0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbcc:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbd2:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0: tdir1 tdir2 tfile3 tfile4 /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0/tdir1: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0/tdir2: /mnt/lustre/.lustre/.ltrash/MDT0000/500: tdir2 tfile4 -u-------------------- /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbd2:0x0 /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbd2:0x0: -u-------------------- /mnt/lustre/.lustre/.ltrash/MDT0000/0 /mnt/lustre/.lustre/.ltrash/MDT0000/0: -u-------------------- /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbcc:0x0 /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbcc:0x0: -u-------------------- /mnt/lustre/.lustre/.ltrash/MDT0000/500 /mnt/lustre/.lustre/.ltrash/MDT0000/500: -u-------------------- /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0 /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0: -u------------e------- /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0/tfile3 -u-------------------- /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0/tdir1 /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0/tdir1: -u------------e------- /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0/tfile4 -u-------------------- /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0/tdir2 /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0/tdir2: -u-------------------- /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe3:0x0 /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe3:0x0: mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117f (8s) == sanityn test 117g: Access .Trash for a non-striped directory ========================================================== 23:33:14 (1779075194) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain /mnt/lustre/.lustre/.ltrash/MDT0000: 0 0x200000bd1:0xbcc:0x0 0x200000bd1:0xbd2:0x0 0x200000bd1:0xbe3:0x0 0x200000bd1:0xbe7:0x0 0x200000bd1:0xbec:0x0 500 /mnt/lustre/.lustre/.ltrash/MDT0000/0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbcc:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbd2:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbec:0x0: tdir1 tfile3 /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbec:0x0/tdir1: /mnt/lustre/.lustre/.ltrash/MDT0000/500: ==> check '.Trash': tdir1 tfile3 <== tdir1 tfile3 tdir1 tfile3 tdir21 tfile22 tdir2 tdir21 tfile22 mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117g (7s) == sanityn test 117h: Permission testing of various operations for files in Trash Can ========================================================== 23:33:21 (1779075201) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain Trash list: ===> /mnt/lustre/.lustre/.ltrash/MDT0000: 0 0x200000bd1:0xbcc:0x0 0x200000bd1:0xbd2:0x0 0x200000bd1:0xbe3:0x0 0x200000bd1:0xbe7:0x0 0x200000bd1:0xbec:0x0 0x200000bd1:0xbf3:0x0 500 /mnt/lustre/.lustre/.ltrash/MDT0000/0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbcc:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbd2:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbec:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0: tdir1 tfile3 /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0/tdir1: /mnt/lustre/.lustre/.ltrash/MDT0000/500: <=== touch: setting times of '/mnt/lustre/.lustre/.ltrash/f117h.sanityn.crfail': No such file or directory mkdir: cannot create directory '/mnt/lustre/.lustre/.ltrash/f117h.sanityn.crfail': Operation not permitted touch: setting times of '/mnt/lustre/.lustre/.ltrash/MDT0000/f117h.sanityn.crfail': No such file or directory mkdir: cannot create directory '/mnt/lustre/.lustre/.ltrash/MDT0000/f117h.sanityn.crfail': Operation not permitted rmdir: failed to remove '/mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0': Operation not permitted touch: setting times of '/mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0/f117h.sanityn.crfail': No such file or directory mv: cannot move '/mnt/lustre/d117h.sanityn/f117h.sanityn.mvfail' to '/mnt/lustre/.lustre/.ltrash/f117h.sanityn.mvfail': Operation not permitted mv: cannot move '/mnt/lustre/d117h.sanityn/f117h.sanityn.mvfail' to '/mnt/lustre/.lustre/.ltrash/MDT0000/f117h.sanityn.mvfail': Operation not permitted mv: cannot move '/mnt/lustre/d117h.sanityn/f117h.sanityn.mvfail' to '/mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0/f117h.sanityn.mvfail': Operation not permitted mv: cannot move '/mnt/lustre/d117h.sanityn/d117h.sanityn.mvfail' to '/mnt/lustre/.lustre/.ltrash/d117h.sanityn.mvfail': Operation not permitted mv: cannot move '/mnt/lustre/d117h.sanityn/d117h.sanityn.mvfail' to '/mnt/lustre/.lustre/.ltrash/MDT0000/d117h.sanityn.mvfail': Operation not permitted mv: cannot move '/mnt/lustre/d117h.sanityn/d117h.sanityn.mvfail' to '/mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0/d117h.sanityn.mvfail': Operation not permitted chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0//tfile3 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0//tfile3 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0//tfile3 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0//tfile3 chmod: changing permissions of '/mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0//tfile3': Bad file descriptor truncate: cannot truncate '/mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0//tfile3' to length 4096: Bad file descriptor chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0//tdir1 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0//tdir1 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0//tdir1 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0//tdir1 chmod: changing permissions of '/mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0//tdir1': Bad file descriptor setfattr: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0//tfile3: Bad file descriptor setfattr: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0//tfile3: Bad file descriptor setfattr: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0//tdir1: Bad file descriptor setfattr: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0//tdir1: Bad file descriptor open: Bad file descriptor open: Bad file descriptor read: Operation not permitted mmap: Operation not permitted open: Is a directory open: Is a directory mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117h (8s) == sanityn test 117i: Forbid to create a file with name .Trash ========================================================== 23:33:29 (1779075209) mkdir: cannot create directory '/mnt/lustre/d117i.sanityn/.Trash': Invalid argument touch: setting times of '/mnt/lustre/d117i.sanityn/.Trash': No such file or directory open(O_RDWR|O_CREAT): Invalid argument mcreate: cannot create `/mnt/lustre/d117i.sanityn/.Trash' with mode 0100644: Invalid argument ln: failed to create hard link '/mnt/lustre/d117i.sanityn/.Trash' => '/mnt/lustre/d117i.sanityn/f117i.sanityn': Invalid argument ln: failed to create symbolic link '/mnt/lustre/d117i.sanityn/.Trash': Invalid argument mv: cannot move '/mnt/lustre/d117i.sanityn/f117i.sanityn' to a subdirectory of itself, '/mnt/lustre/d117i.sanityn/.Trash' mv: cannot move '/mnt/lustre/d117i.sanityn/f117i.sanityn' to a subdirectory of itself, '/mnt/lustre/.Trash' ln: failed to create symbolic link '/mnt/lustre/d117i.sanityn/.Trash': Invalid argument mv: cannot move '/mnt/lustre/d117i.sanityn/d117i.sanityn' to a subdirectory of itself, '/mnt/lustre/d117i.sanityn/.Trash' mv: cannot move '/mnt/lustre/d117i.sanityn/d117i.sanityn' to a subdirectory of itself, '/mnt/lustre/.Trash' PASS 117i (3s) == sanityn test 117j: Move back file from Trash Can ====== 23:33:32 (1779075212) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain tfile.1 -u------------e------- /mnt/lustre/d117j.sanityn/.Trash/tfile.1 --------------e------- /mnt/lustre/d117j.sanityn/tfile.1 List /mnt/lustre/d117j.sanityn/.Trash /mnt/lustre/d117j.sanityn/.Trash: tdir1 /mnt/lustre/d117j.sanityn/.Trash/tdir1: tdir11 tfile12 tfile13 /mnt/lustre/d117j.sanityn/.Trash/tdir1/tdir11: -u-------------------- /mnt/lustre/d117j.sanityn/.Trash/tdir1 /mnt/lustre/d117j.sanityn/.Trash/tdir1: -u------------e------- /mnt/lustre/d117j.sanityn/.Trash/tdir1/tfile12 -u-------------------- /mnt/lustre/d117j.sanityn/.Trash/tdir1/tdir11 /mnt/lustre/d117j.sanityn/.Trash/tdir1/tdir11: -u------------e------- /mnt/lustre/d117j.sanityn/.Trash/tdir1/tfile13 === DONE === mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117j (7s) == sanityn test 117k: Unrm file from Trash Can =========== 23:33:39 (1779075219) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain ==== DONE ==== mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117k (8s) == sanityn test 117l: Clean up files in Trash Can via command 'lfs trash clean' ========================================================== 23:33:47 (1779075227) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain Phase 0/PR: remove dir 1:0x200000bd1:0xbd2:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xbd2:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xbd2:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xbd2:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xc12:0x0 from Trash Can Phase 0/PR: remove dir 2:tdir1 from Trash Can Phase 1/EN: remove file 3:tfile12 from Trash Can Phase 0/PR: remove dir 3:tdir11 from Trash Can Phase 2/PO: remove dir 3:tdir11 from Trash Can Phase 1/EN: remove dir 3:tdir11 from Trash Can Phase 1/EN: remove file 3:tfile13 from Trash Can Phase 2/PO: remove dir 2:tdir1 from Trash Can Phase 1/EN: remove dir 2:tdir1 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xc12:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xc12:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xc12:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xc06:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xc06:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xc06:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xc06:0x0 from Trash Can Phase 0/PR: remove dir 1:0 from Trash Can Phase 2/PO: remove dir 1:0 from Trash Can Phase 1/EN: remove dir 1:0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xbcc:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xbcc:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xbcc:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xbcc:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xbec:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xbec:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xbec:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xbec:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xc0c:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xc0c:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xc0c:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xc0c:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xbf3:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xbf3:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xbf3:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xbf3:0x0 from Trash Can Phase 0/PR: remove dir 1:500 from Trash Can Phase 2/PO: remove dir 1:500 from Trash Can Phase 1/EN: remove dir 1:500 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xbe7:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xbe7:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xbe7:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xbe7:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xbe3:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xbe3:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xbe3:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xbe3:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xbd2:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xbd2:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xbd2:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xbd2:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xc12:0x0 from Trash Can Phase 0/PR: remove dir 2:tdir1 from Trash Can Phase 1/EN: remove file 3:tfile12 from Trash Can Phase 0/PR: remove dir 3:tdir11 from Trash Can Phase 2/PO: remove dir 3:tdir11 from Trash Can Phase 1/EN: remove dir 3:tdir11 from Trash Can Phase 1/EN: remove file 3:tfile13 from Trash Can Phase 2/PO: remove dir 2:tdir1 from Trash Can Phase 1/EN: remove dir 2:tdir1 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xc12:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xc12:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xc12:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xc06:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xc06:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xc06:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xc06:0x0 from Trash Can Phase 0/PR: remove dir 1:0 from Trash Can Phase 2/PO: remove dir 1:0 from Trash Can Phase 1/EN: remove dir 1:0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xbcc:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xbcc:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xbcc:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xbcc:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xbec:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xbec:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xbec:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xbec:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xc0c:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xc0c:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xc0c:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xc0c:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xbf3:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xbf3:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xbf3:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xbf3:0x0 from Trash Can Phase 0/PR: remove dir 1:500 from Trash Can Phase 2/PO: remove dir 1:500 from Trash Can Phase 1/EN: remove dir 1:500 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xc1c:0x0 from Trash Can Phase 1/EN: remove file 2:tfile22 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xc1c:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xc1c:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xc1c:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xbe7:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xbe7:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xbe7:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xbe7:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xbe3:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xbe3:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xbe3:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xbe3:0x0 from Trash Can mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117l (7s) == sanityn test 117m: Empty Trash Can via the command 'lfs trash clean' ========================================================== 23:33:54 (1779075234) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain /mnt/lustre/.lustre/.ltrash: MDT0000 MDT0001 /mnt/lustre/.lustre/.ltrash/MDT0000: 0 0x200000bd1:0xbcc:0x0 0x200000bd1:0xbd2:0x0 0x200000bd1:0xbe3:0x0 0x200000bd1:0xbe7:0x0 0x200000bd1:0xbec:0x0 0x200000bd1:0xbf3:0x0 0x200000bd1:0xc06:0x0 0x200000bd1:0xc0c:0x0 0x200000bd1:0xc12:0x0 0x200000bd1:0xc1c:0x0 0x200000bd1:0xc1f:0x0 0x200000bd1:0xc25:0x0 500 /mnt/lustre/.lustre/.ltrash/MDT0000/0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbcc:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbd2:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbec:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc06:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc0c:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc12:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc1c:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc1f:0x0: tdir1 /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc1f:0x0/tdir1: tdir11 tfile12 tfile13 /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc1f:0x0/tdir1/tdir11: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc25:0x0: tfile22 /mnt/lustre/.lustre/.ltrash/MDT0000/500: /mnt/lustre/.lustre/.ltrash/MDT0001: Phase 0/PR: remove dir 1:0x200000bd1:0xbd2:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xbd2:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xbd2:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xbd2:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xc12:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xc12:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xc12:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xc12:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xc06:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xc06:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xc06:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xc06:0x0 from Trash Can Phase 0/PR: remove dir 1:0 from Trash Can Phase 2/PO: remove dir 1:0 from Trash Can Phase 1/EN: remove dir 1:0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xc25:0x0 from Trash Can Phase 1/EN: remove file 2:tfile22 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xc25:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xc25:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xc25:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xbcc:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xbcc:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xbcc:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xbcc:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xbec:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xbec:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xbec:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xbec:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xc1f:0x0 from Trash Can Phase 0/PR: remove dir 2:tdir1 from Trash Can Phase 1/EN: remove file 3:tfile12 from Trash Can Phase 0/PR: remove dir 3:tdir11 from Trash Can Phase 2/PO: remove dir 3:tdir11 from Trash Can Phase 1/EN: remove dir 3:tdir11 from Trash Can Phase 1/EN: remove file 3:tfile13 from Trash Can Phase 2/PO: remove dir 2:tdir1 from Trash Can Phase 1/EN: remove dir 2:tdir1 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xc1f:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xc1f:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xc1f:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xc0c:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xc0c:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xc0c:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xc0c:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xbf3:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xbf3:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xbf3:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xbf3:0x0 from Trash Can Phase 0/PR: remove dir 1:500 from Trash Can Phase 2/PO: remove dir 1:500 from Trash Can Phase 1/EN: remove dir 1:500 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xc1c:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xc1c:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xc1c:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xc1c:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xbe7:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xbe7:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xbe7:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xbe7:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xbe3:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xbe3:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xbe3:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xbe3:0x0 from Trash Can mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117m (7s) == sanityn test 117n: Update LinkEA accordingly when moving file into Trash Can ========================================================== 23:34:01 (1779075241) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain /mnt/lustre/d117n.sanityn/.Trash//tfile.1: count 1 0: pfid [0x200000bd0:0x652:0x0], name 'tfile.1' /mnt/lustre/d117n.sanityn/.Trash//tfile.1: count 1 0: pfid [0x200000bd0:0x652:0x0], name 'tfile.1' /mnt/lustre/d117n.sanityn/.Trash//tfile.1: count 1 0: pfid [0x200000bd0:0x652:0x0], name 'tfile.1' /mnt/lustre/d117n.sanityn/.Trash//tfile.1: count 1 0: pfid [0x200000bd0:0x652:0x0], name 'tfile.1' /mnt/lustre/d117n.sanityn/.Trash//tdir1/tdir11: count 1 0: pfid [0x200000bd0:0x653:0x0], name 'tdir11' /mnt/lustre/d117n.sanityn/.Trash//tdir1/tdir11: count 1 0: pfid [0x200000bd0:0x653:0x0], name 'tdir11' /mnt/lustre/d117n.sanityn/.Trash//tdir1/tdir11: count 1 0: pfid [0x200000bd0:0x653:0x0], name 'tdir11' /mnt/lustre/d117n.sanityn/.Trash//tdir1/tdir11: count 1 0: pfid [0x200000bd0:0x653:0x0], name 'tdir11' /mnt/lustre/d117n.sanityn/.Trash//tdir1/tfile12: count 1 0: pfid [0x200000bd0:0x653:0x0], name 'tfile12' /mnt/lustre/d117n.sanityn/.Trash//tdir1/tfile12: count 1 0: pfid [0x200000bd0:0x653:0x0], name 'tfile12' /mnt/lustre/d117n.sanityn/.Trash//tdir1/tfile12: count 1 0: pfid [0x200000bd0:0x653:0x0], name 'tfile12' /mnt/lustre/d117n.sanityn/.Trash//tdir1/tfile12: count 1 0: pfid [0x200000bd0:0x653:0x0], name 'tfile12' /mnt/lustre/d117n.sanityn/.Trash//tdir1/tfile13: count 1 0: pfid [0x200000bd0:0x653:0x0], name 'tfile13' /mnt/lustre/d117n.sanityn/.Trash//tdir1/tfile13: count 1 0: pfid [0x200000bd0:0x653:0x0], name 'tfile13' /mnt/lustre/d117n.sanityn/.Trash//tdir1/tfile13: count 1 0: pfid [0x200000bd0:0x653:0x0], name 'tfile13' /mnt/lustre/d117n.sanityn/.Trash//tdir1/tfile13: count 1 0: pfid [0x200000bd0:0x653:0x0], name 'tfile13' /mnt/lustre/d117n.sanityn/.Trash//tdir1: count 1 0: pfid [0x200000bd0:0x652:0x0], name 'tdir1' /mnt/lustre/d117n.sanityn/.Trash//tdir1: count 1 0: pfid [0x200000bd0:0x652:0x0], name 'tdir1' /mnt/lustre/d117n.sanityn/.Trash//tdir1: count 1 0: pfid [0x200000bd0:0x652:0x0], name 'tdir1' /mnt/lustre/d117n.sanityn/.Trash//tdir1: count 1 0: pfid [0x200000bd0:0x652:0x0], name 'tdir1' /mnt/lustre/d117n.sanityn/tdir2/.Trash//tfile22: count 1 0: pfid [0x200000bd0:0x654:0x0], name 'tfile22' /mnt/lustre/d117n.sanityn/tdir2/.Trash//tfile22: count 1 0: pfid [0x200000bd0:0x654:0x0], name 'tfile22' /mnt/lustre/d117n.sanityn/tdir2/.Trash//tfile22: count 1 0: pfid [0x200000bd0:0x654:0x0], name 'tfile22' /mnt/lustre/d117n.sanityn/tdir2/.Trash//tfile22: count 1 0: pfid [0x200000bd0:0x654:0x0], name 'tfile22' mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117n (7s) == sanityn test 117o: Set/Clear NOTRASH flag on file ===== 23:34:08 (1779075248) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain Set flags 0X80000/0X80040 OK: rc=0 file: /mnt/lustre/d117o.sanityn/tfile.1, flags: 0X702DDCD200080040 notrash file: /mnt/lustre/d117o.sanityn/tfile.1, flags: 0XB1FACE6F00080040 notrash file: /mnt/lustre/d117o.sanityn/tdir1, flags: 0XA6D627FF00000040 notrash file: /mnt/lustre/d117o.sanityn/tdir1/tdir11, flags: 0X9C51D5FB00000040 notrash file: /mnt/lustre/d117o.sanityn/tdir1, flags: 0XA391621800000040 notrash file: /mnt/lustre/d117o.sanityn/tdir1/tdir11, flags: 0XA391621800000040 notrash file: /mnt/lustre/d117o.sanityn/tdir1/tfile12, flags: 0XA391621800080040 notrash lsattr /mnt/lustre/d117o.sanityn/tdir1: ------d-------e------- /mnt/lustre/d117o.sanityn/tdir1/tfile12 ------d--------------- /mnt/lustre/d117o.sanityn/tdir1/tdir11 lsattr /mnt/lustre/d117o.sanityn/tdir1/tdir11: lsattr /mnt/lustre/d117o.sanityn/tdir1/tfile12: ------d-------e------- /mnt/lustre/d117o.sanityn/tdir1/tfile12 file: /mnt/lustre/d117o.sanityn/tdir1/tdir11, flags: 0XF47A633700000040 notrash file: /mnt/lustre/d117o.sanityn/tdir1/tfile12, flags: 0X2ED9A1F900080040 notrash file: /mnt/lustre/d117o.sanityn/tdir1/tfile13, flags: 0X3B8AE79100080040 notrash Set flags 0X80040/0X80000 OK: rc=0 chattr 1.47.3-wc2 (11-Nov-2025) Flags of /mnt/lustre/d117o.sanityn/tdir1 set as ---------------------- file: /mnt/lustre/d117o.sanityn/tdir1/tfile13, flags: 0X904570D700080000 tfile13 mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117o (7s) == sanityn test 117p: Trash Can handling for remote dir/file ========================================================== 23:34:15 (1779075255) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain /mnt/lustre/d117p.sanityn/d117p.sanityn.1/.Trash/f117p.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6165 0x1815 0x280000401 /mnt/lustre/.lustre/.ltrash: MDT0000 MDT0001 /mnt/lustre/.lustre/.ltrash/MDT0000: 0 0x200000bd1:0xbcc:0x0 0x200000bd1:0xbd2:0x0 0x200000bd1:0xbe3:0x0 0x200000bd1:0xbe7:0x0 0x200000bd1:0xbec:0x0 0x200000bd1:0xbf3:0x0 0x200000bd1:0xc06:0x0 0x200000bd1:0xc0c:0x0 0x200000bd1:0xc12:0x0 0x200000bd1:0xc1c:0x0 0x200000bd1:0xc1f:0x0 0x200000bd1:0xc25:0x0 0x200000bd1:0xc28:0x0 0x200000bd1:0xc2e:0x0 0x200000bd1:0xc33:0x0 500 /mnt/lustre/.lustre/.ltrash/MDT0000/0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbcc:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbd2:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbec:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc06:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc0c:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc12:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc1c:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc1f:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc25:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc28:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc2e:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc33:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/500: /mnt/lustre/.lustre/.ltrash/MDT0001: 0x240000405:0xc:0x0 /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000405:0xc:0x0: f117p.sanityn /mnt/lustre/.lustre/.ltrash: MDT0000 MDT0001 /mnt/lustre/.lustre/.ltrash/MDT0000: 0 0x200000bd1:0xbcc:0x0 0x200000bd1:0xbd2:0x0 0x200000bd1:0xbe3:0x0 0x200000bd1:0xbe7:0x0 0x200000bd1:0xbec:0x0 0x200000bd1:0xbf3:0x0 0x200000bd1:0xc06:0x0 0x200000bd1:0xc0c:0x0 0x200000bd1:0xc12:0x0 0x200000bd1:0xc1c:0x0 0x200000bd1:0xc1f:0x0 0x200000bd1:0xc25:0x0 0x200000bd1:0xc28:0x0 0x200000bd1:0xc2e:0x0 0x200000bd1:0xc33:0x0 0x200000bd1:0xc37:0x0 500 /mnt/lustre/.lustre/.ltrash/MDT0000/0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbcc:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbd2:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbec:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc06:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc0c:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc12:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc1c:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc1f:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc25:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc28:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc2e:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc33:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc37:0x0: d117p.sanityn.01 /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc37:0x0/d117p.sanityn.01: /mnt/lustre/.lustre/.ltrash/MDT0000/500: /mnt/lustre/.lustre/.ltrash/MDT0001: 0x240000405:0xc:0x0 /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000405:0xc:0x0: f117p.sanityn mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117p (8s) == sanityn test 117pb: Trash Can handling for remote dir/file with multiple levels ========================================================== 23:34:23 (1779075263) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain /mnt/lustre/.lustre/.ltrash: MDT0000 MDT0001 /mnt/lustre/.lustre/.ltrash/MDT0000: 0 0x200000bd1:0xbcc:0x0 0x200000bd1:0xbd2:0x0 0x200000bd1:0xbe3:0x0 0x200000bd1:0xbe7:0x0 0x200000bd1:0xbec:0x0 0x200000bd1:0xbf3:0x0 0x200000bd1:0xc06:0x0 0x200000bd1:0xc0c:0x0 0x200000bd1:0xc12:0x0 0x200000bd1:0xc1c:0x0 0x200000bd1:0xc1f:0x0 0x200000bd1:0xc25:0x0 0x200000bd1:0xc28:0x0 0x200000bd1:0xc2e:0x0 0x200000bd1:0xc33:0x0 0x200000bd1:0xc37:0x0 0x200000bd1:0xc39:0x0 500 /mnt/lustre/.lustre/.ltrash/MDT0000/0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbcc:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbd2:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbec:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc06:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc0c:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc12:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc1c:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc1f:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc25:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc28:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc2e:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc33:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc37:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc39:0x0: d117pb.sanityn.0 /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc39:0x0/d117pb.sanityn.0: d117pb.sanityn.01 f117pb.sanityn.02 /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc39:0x0/d117pb.sanityn.0/d117pb.sanityn.01: /mnt/lustre/.lustre/.ltrash/MDT0000/500: /mnt/lustre/.lustre/.ltrash/MDT0001: 0x240000405:0xc:0x0 /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000405:0xc:0x0: mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117pb (7s) == sanityn test 117q: Access Trash Can via '.Trash' for striped directory ========================================================== 23:34:30 (1779075270) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain #0 -> 0x200000bd0:0x657:0x0 #1 -> 0x240000404:0x5af:0x0 file.1 file.2 Creating 100 files under striped directory: /mnt/lustre/d117q.sanityn/striped_dir total: 100 open/close in 0.45 seconds: 221.19 ops/second f117q.sanityn0 -> 0 f117q.sanityn1 -> 0 f117q.sanityn2 -> 0 f117q.sanityn3 -> 0 f117q.sanityn4 -> 0 f117q.sanityn5 -> 0 f117q.sanityn6 -> 0 f117q.sanityn7 -> 0 f117q.sanityn8 -> 0 f117q.sanityn9 -> 0 f117q.sanityn10 -> 0 f117q.sanityn11 -> 0 f117q.sanityn12 -> 0 f117q.sanityn13 -> 0 f117q.sanityn14 -> 1 f117q.sanityn15 -> 1 f117q.sanityn16 -> 0 f117q.sanityn17 -> 1 f117q.sanityn18 -> 0 f117q.sanityn19 -> 1 f117q.sanityn20 -> 1 f117q.sanityn21 -> 1 f117q.sanityn22 -> 1 f117q.sanityn23 -> 1 f117q.sanityn24 -> 1 f117q.sanityn25 -> 0 f117q.sanityn26 -> 1 f117q.sanityn27 -> 0 f117q.sanityn28 -> 1 f117q.sanityn29 -> 1 f117q.sanityn30 -> 1 f117q.sanityn31 -> 1 f117q.sanityn32 -> 0 f117q.sanityn33 -> 1 f117q.sanityn34 -> 0 f117q.sanityn35 -> 0 f117q.sanityn36 -> 1 f117q.sanityn37 -> 0 f117q.sanityn38 -> 1 f117q.sanityn39 -> 0 f117q.sanityn40 -> 1 f117q.sanityn41 -> 0 f117q.sanityn42 -> 1 f117q.sanityn43 -> 1 f117q.sanityn44 -> 0 f117q.sanityn45 -> 0 f117q.sanityn46 -> 0 f117q.sanityn47 -> 1 f117q.sanityn48 -> 1 f117q.sanityn49 -> 0 f117q.sanityn50 -> 0 f117q.sanityn51 -> 0 f117q.sanityn52 -> 1 f117q.sanityn53 -> 1 f117q.sanityn54 -> 1 f117q.sanityn55 -> 0 f117q.sanityn56 -> 1 f117q.sanityn57 -> 1 f117q.sanityn58 -> 1 f117q.sanityn59 -> 0 f117q.sanityn60 -> 1 f117q.sanityn61 -> 1 f117q.sanityn62 -> 1 f117q.sanityn63 -> 1 f117q.sanityn64 -> 1 f117q.sanityn65 -> 0 f117q.sanityn66 -> 1 f117q.sanityn67 -> 1 f117q.sanityn68 -> 0 f117q.sanityn69 -> 0 f117q.sanityn70 -> 0 f117q.sanityn71 -> 0 f117q.sanityn72 -> 0 f117q.sanityn73 -> 0 f117q.sanityn74 -> 0 f117q.sanityn75 -> 0 f117q.sanityn76 -> 1 f117q.sanityn77 -> 0 f117q.sanityn78 -> 1 f117q.sanityn79 -> 0 f117q.sanityn80 -> 1 f117q.sanityn81 -> 1 f117q.sanityn82 -> 0 f117q.sanityn83 -> 0 f117q.sanityn84 -> 1 f117q.sanityn85 -> 0 f117q.sanityn86 -> 1 f117q.sanityn87 -> 1 f117q.sanityn88 -> 1 f117q.sanityn89 -> 0 f117q.sanityn90 -> 1 f117q.sanityn91 -> 0 f117q.sanityn92 -> 0 f117q.sanityn93 -> 1 f117q.sanityn94 -> 0 f117q.sanityn95 -> 1 f117q.sanityn96 -> 0 f117q.sanityn97 -> 1 f117q.sanityn98 -> 0 f117q.sanityn99 -> 0 - unlinked 0 (time 1779075275 ; total 0 ; last 0) total: 100 unlinks in 0 seconds: inf unlinks/second f117q.sanityn0 f117q.sanityn1 f117q.sanityn10 f117q.sanityn11 f117q.sanityn12 f117q.sanityn13 f117q.sanityn14 f117q.sanityn15 f117q.sanityn16 f117q.sanityn17 f117q.sanityn18 f117q.sanityn19 f117q.sanityn2 f117q.sanityn20 f117q.sanityn21 f117q.sanityn22 f117q.sanityn23 f117q.sanityn24 f117q.sanityn25 f117q.sanityn26 f117q.sanityn27 f117q.sanityn28 f117q.sanityn29 f117q.sanityn3 f117q.sanityn30 f117q.sanityn31 f117q.sanityn32 f117q.sanityn33 f117q.sanityn34 f117q.sanityn35 f117q.sanityn36 f117q.sanityn37 f117q.sanityn38 f117q.sanityn39 f117q.sanityn4 f117q.sanityn40 f117q.sanityn41 f117q.sanityn42 f117q.sanityn43 f117q.sanityn44 f117q.sanityn45 f117q.sanityn46 f117q.sanityn47 f117q.sanityn48 f117q.sanityn49 f117q.sanityn5 f117q.sanityn50 f117q.sanityn51 f117q.sanityn52 f117q.sanityn53 f117q.sanityn54 f117q.sanityn55 f117q.sanityn56 f117q.sanityn57 f117q.sanityn58 f117q.sanityn59 f117q.sanityn6 f117q.sanityn60 f117q.sanityn61 f117q.sanityn62 f117q.sanityn63 f117q.sanityn64 f117q.sanityn65 f117q.sanityn66 f117q.sanityn67 f117q.sanityn68 f117q.sanityn69 f117q.sanityn7 f117q.sanityn70 f117q.sanityn71 f117q.sanityn72 f117q.sanityn73 f117q.sanityn74 f117q.sanityn75 f117q.sanityn76 f117q.sanityn77 f117q.sanityn78 f117q.sanityn79 f117q.sanityn8 f117q.sanityn80 f117q.sanityn81 f117q.sanityn82 f117q.sanityn83 f117q.sanityn84 f117q.sanityn85 f117q.sanityn86 f117q.sanityn87 f117q.sanityn88 f117q.sanityn89 f117q.sanityn9 f117q.sanityn90 f117q.sanityn91 f117q.sanityn92 f117q.sanityn93 f117q.sanityn94 f117q.sanityn95 f117q.sanityn96 f117q.sanityn97 f117q.sanityn98 f117q.sanityn99 f117q.sanityn10 f117q.sanityn11 f117q.sanityn12 f117q.sanityn13 f117q.sanityn14 f117q.sanityn15 f117q.sanityn16 f117q.sanityn17 f117q.sanityn18 f117q.sanityn19 f117q.sanityn20 f117q.sanityn21 f117q.sanityn22 f117q.sanityn23 f117q.sanityn24 f117q.sanityn25 f117q.sanityn26 f117q.sanityn27 f117q.sanityn28 f117q.sanityn29 f117q.sanityn30 f117q.sanityn31 f117q.sanityn32 f117q.sanityn33 f117q.sanityn34 f117q.sanityn35 f117q.sanityn36 f117q.sanityn37 f117q.sanityn38 f117q.sanityn39 f117q.sanityn40 f117q.sanityn41 f117q.sanityn42 f117q.sanityn43 f117q.sanityn44 f117q.sanityn45 f117q.sanityn46 f117q.sanityn47 f117q.sanityn48 f117q.sanityn49 f117q.sanityn50 f117q.sanityn51 f117q.sanityn52 f117q.sanityn53 f117q.sanityn54 f117q.sanityn55 f117q.sanityn56 f117q.sanityn57 f117q.sanityn58 f117q.sanityn59 f117q.sanityn60 f117q.sanityn61 f117q.sanityn62 f117q.sanityn63 f117q.sanityn64 f117q.sanityn65 f117q.sanityn66 f117q.sanityn67 f117q.sanityn68 f117q.sanityn69 f117q.sanityn70 f117q.sanityn71 f117q.sanityn72 f117q.sanityn73 f117q.sanityn74 f117q.sanityn75 f117q.sanityn76 f117q.sanityn77 f117q.sanityn78 f117q.sanityn79 f117q.sanityn80 f117q.sanityn81 f117q.sanityn82 f117q.sanityn83 f117q.sanityn84 f117q.sanityn85 f117q.sanityn86 f117q.sanityn87 f117q.sanityn88 f117q.sanityn89 f117q.sanityn90 f117q.sanityn91 f117q.sanityn92 f117q.sanityn93 f117q.sanityn94 f117q.sanityn95 f117q.sanityn96 f117q.sanityn97 f117q.sanityn98 f117q.sanityn99 mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117q (13s) == sanityn test 117r: Verify the deleting a striped directory on Trash Can ========================================================== 23:34:43 (1779075283) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain #0 -> 0x200000bd0:0x659:0x0 #1 -> 0x240000404:0x5b0:0x0 Creating 50 files under striped directory: /mnt/lustre/d117r.sanityn/striped_dir total: 50 open/close in 0.24 seconds: 212.58 ops/second f117r.sanityn0 -> 0 f117r.sanityn1 -> 0 f117r.sanityn2 -> 0 f117r.sanityn3 -> 0 f117r.sanityn4 -> 0 f117r.sanityn5 -> 0 f117r.sanityn6 -> 0 f117r.sanityn7 -> 0 f117r.sanityn8 -> 0 f117r.sanityn9 -> 0 f117r.sanityn10 -> 1 f117r.sanityn11 -> 1 f117r.sanityn12 -> 1 f117r.sanityn13 -> 1 f117r.sanityn14 -> 1 f117r.sanityn15 -> 0 f117r.sanityn16 -> 1 f117r.sanityn17 -> 1 f117r.sanityn18 -> 0 f117r.sanityn19 -> 1 f117r.sanityn20 -> 0 f117r.sanityn21 -> 0 f117r.sanityn22 -> 0 f117r.sanityn23 -> 0 f117r.sanityn24 -> 0 f117r.sanityn25 -> 1 f117r.sanityn26 -> 0 f117r.sanityn27 -> 1 f117r.sanityn28 -> 1 f117r.sanityn29 -> 1 f117r.sanityn30 -> 1 f117r.sanityn31 -> 0 f117r.sanityn32 -> 0 f117r.sanityn33 -> 0 f117r.sanityn34 -> 1 f117r.sanityn35 -> 0 f117r.sanityn36 -> 0 f117r.sanityn37 -> 0 f117r.sanityn38 -> 0 f117r.sanityn39 -> 1 f117r.sanityn40 -> 0 f117r.sanityn41 -> 1 f117r.sanityn42 -> 0 f117r.sanityn43 -> 1 f117r.sanityn44 -> 0 f117r.sanityn45 -> 1 f117r.sanityn46 -> 0 f117r.sanityn47 -> 0 f117r.sanityn48 -> 1 f117r.sanityn49 -> 0 - unlinked 0 (time 1779075287 ; total 0 ; last 0) total: 50 unlinks in 0 seconds: inf unlinks/second Check files in Trash Can: f117r.sanityn0 f117r.sanityn1 f117r.sanityn10 f117r.sanityn11 f117r.sanityn12 f117r.sanityn13 f117r.sanityn14 f117r.sanityn15 f117r.sanityn16 f117r.sanityn17 f117r.sanityn18 f117r.sanityn19 f117r.sanityn2 f117r.sanityn20 f117r.sanityn21 f117r.sanityn22 f117r.sanityn23 f117r.sanityn24 f117r.sanityn25 f117r.sanityn26 f117r.sanityn27 f117r.sanityn28 f117r.sanityn29 f117r.sanityn3 f117r.sanityn30 f117r.sanityn31 f117r.sanityn32 f117r.sanityn33 f117r.sanityn34 f117r.sanityn35 f117r.sanityn36 f117r.sanityn37 f117r.sanityn38 f117r.sanityn39 f117r.sanityn4 f117r.sanityn40 f117r.sanityn41 f117r.sanityn42 f117r.sanityn43 f117r.sanityn44 f117r.sanityn45 f117r.sanityn46 f117r.sanityn47 f117r.sanityn48 f117r.sanityn49 f117r.sanityn5 f117r.sanityn6 f117r.sanityn7 f117r.sanityn8 f117r.sanityn9 f117r.sanityn10 f117r.sanityn11 f117r.sanityn12 f117r.sanityn13 f117r.sanityn14 f117r.sanityn15 f117r.sanityn16 f117r.sanityn17 f117r.sanityn18 f117r.sanityn19 f117r.sanityn20 f117r.sanityn21 f117r.sanityn22 f117r.sanityn23 f117r.sanityn24 f117r.sanityn25 f117r.sanityn26 f117r.sanityn27 f117r.sanityn28 f117r.sanityn29 f117r.sanityn30 f117r.sanityn31 f117r.sanityn32 f117r.sanityn33 f117r.sanityn34 f117r.sanityn35 f117r.sanityn36 f117r.sanityn37 f117r.sanityn38 f117r.sanityn39 f117r.sanityn40 f117r.sanityn41 f117r.sanityn42 f117r.sanityn43 f117r.sanityn44 f117r.sanityn45 f117r.sanityn46 f117r.sanityn47 f117r.sanityn48 f117r.sanityn49 FID for /mnt/lustre/d117r.sanityn/striped_dir [0x200000bd1:0xc72:0x0] pfid[0]=0x200000bd0:0x659:0x0 [0x200000400:0x2:0x0] [0x200000bd0:0x65a:0x0] pfid[1]=0x240000404:0x5b0:0x0 [0x240000400:0x2:0x0] [0x240000400:0xcd2:0x0] Obtain LMVEA for virtual dir /mnt/lustre/d117r.sanityn/striped_dir/.Trash: lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x200000bd0:0x65a:0x0] 1 [0x240000400:0xcd2:0x0] #0 -> name: 0x200000bd0:0x659:0x0 fid: 0x200000bd0:0x65a:0x0 #1 -> name: 0x240000404:0x5b0:0x0 fid: 0x240000400:0xcd2:0x0 /mnt/lustre/.lustre/.ltrash: MDT0000 MDT0001 /mnt/lustre/.lustre/.ltrash/MDT0000: 0 0x200000bd0:0x657:0x0 0x200000bd1:0xbcc:0x0 0x200000bd1:0xbd2:0x0 0x200000bd1:0xbe3:0x0 0x200000bd1:0xbe7:0x0 0x200000bd1:0xbec:0x0 0x200000bd1:0xbf3:0x0 0x200000bd1:0xc06:0x0 0x200000bd1:0xc0c:0x0 0x200000bd1:0xc12:0x0 0x200000bd1:0xc1c:0x0 0x200000bd1:0xc1f:0x0 0x200000bd1:0xc25:0x0 0x200000bd1:0xc28:0x0 0x200000bd1:0xc2e:0x0 0x200000bd1:0xc33:0x0 0x200000bd1:0xc37:0x0 0x200000bd1:0xc39:0x0 500 /mnt/lustre/.lustre/.ltrash/MDT0000/0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd0:0x657:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbcc:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbd2:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbec:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc06:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc0c:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc12:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc1c:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc1f:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc25:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc28:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc2e:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc33:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc37:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc39:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/500: /mnt/lustre/.lustre/.ltrash/MDT0001: 0x240000404:0x5af:0x0 0x240000405:0x41:0x0 0x240000405:0xc:0x0 /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000404:0x5af:0x0: /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000405:0x41:0x0: striped_dir /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000405:0x41:0x0/striped_dir: f117r.sanityn0 f117r.sanityn1 f117r.sanityn10 f117r.sanityn11 f117r.sanityn12 f117r.sanityn13 f117r.sanityn14 f117r.sanityn15 f117r.sanityn16 f117r.sanityn17 f117r.sanityn18 f117r.sanityn19 f117r.sanityn2 f117r.sanityn20 f117r.sanityn21 f117r.sanityn22 f117r.sanityn23 f117r.sanityn24 f117r.sanityn25 f117r.sanityn26 f117r.sanityn27 f117r.sanityn28 f117r.sanityn29 f117r.sanityn3 f117r.sanityn30 f117r.sanityn31 f117r.sanityn32 f117r.sanityn33 f117r.sanityn34 f117r.sanityn35 f117r.sanityn36 f117r.sanityn37 f117r.sanityn38 f117r.sanityn39 f117r.sanityn4 f117r.sanityn40 f117r.sanityn41 f117r.sanityn42 f117r.sanityn43 f117r.sanityn44 f117r.sanityn45 f117r.sanityn46 f117r.sanityn47 f117r.sanityn48 f117r.sanityn49 f117r.sanityn5 f117r.sanityn6 f117r.sanityn7 f117r.sanityn8 f117r.sanityn9 file.1 file.2 /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000405:0xc:0x0: Obtain LMVEA for striped dir /mnt/lustre/d117r.sanityn/striped_dir: lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x200000bd0:0x65a:0x0] 1 [0x240000400:0xcd2:0x0] [0x200000bd0:0x65c:0x0] #0 -> name: 0x200000bd0:0x659:0x0 fid: 0x200000bd0:0x65a:0x0 #1 -> name: 0x240000404:0x5b0:0x0 fid: 0x240000400:0xcd2:0x0 lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x200000bd0:0x65a:0x0] 1 [0x240000400:0xcd2:0x0] f117r.sanityn0 f117r.sanityn1 f117r.sanityn10 f117r.sanityn11 f117r.sanityn12 f117r.sanityn13 f117r.sanityn14 f117r.sanityn15 f117r.sanityn16 f117r.sanityn17 f117r.sanityn18 f117r.sanityn19 f117r.sanityn2 f117r.sanityn20 f117r.sanityn21 f117r.sanityn22 f117r.sanityn23 f117r.sanityn24 f117r.sanityn25 f117r.sanityn26 f117r.sanityn27 f117r.sanityn28 f117r.sanityn29 f117r.sanityn3 f117r.sanityn30 f117r.sanityn31 f117r.sanityn32 f117r.sanityn33 f117r.sanityn34 f117r.sanityn35 f117r.sanityn36 f117r.sanityn37 f117r.sanityn38 f117r.sanityn39 f117r.sanityn4 f117r.sanityn40 f117r.sanityn41 f117r.sanityn42 f117r.sanityn43 f117r.sanityn44 f117r.sanityn45 f117r.sanityn46 f117r.sanityn47 f117r.sanityn48 f117r.sanityn49 f117r.sanityn5 f117r.sanityn6 f117r.sanityn7 f117r.sanityn8 f117r.sanityn9 file.1 file.2 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn0, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn1, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn10, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn11, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn12, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn13, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn14, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn15, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn16, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn17, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn18, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn19, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn2, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn20, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn21, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn22, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn23, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn24, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn25, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn26, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn27, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn28, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn29, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn3, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn30, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn31, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn32, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn33, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn34, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn35, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn36, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn37, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn38, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn39, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn4, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn40, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn41, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn42, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn43, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn44, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn45, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn46, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn47, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn48, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn49, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn5, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn6, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn7, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn8, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/f117r.sanityn9, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/file.1, flags: 0X740F155F00080000 file: /mnt/lustre/d117r.sanityn/striped_dir/file.2, flags: 0X740F155F00080000 mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117r (12s) == sanityn test 117s: Create trash shard if it does not exist ========================================================== 23:34:55 (1779075295) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain /mnt/lustre/.lustre/.ltrash: MDT0000 MDT0001 /mnt/lustre/.lustre/.ltrash/MDT0000: 0 0x200000bd0:0x657:0x0 0x200000bd1:0xbcc:0x0 0x200000bd1:0xbd2:0x0 0x200000bd1:0xbe3:0x0 0x200000bd1:0xbe7:0x0 0x200000bd1:0xbec:0x0 0x200000bd1:0xbf3:0x0 0x200000bd1:0xc06:0x0 0x200000bd1:0xc0c:0x0 0x200000bd1:0xc12:0x0 0x200000bd1:0xc1c:0x0 0x200000bd1:0xc1f:0x0 0x200000bd1:0xc25:0x0 0x200000bd1:0xc28:0x0 0x200000bd1:0xc2e:0x0 0x200000bd1:0xc33:0x0 0x200000bd1:0xc37:0x0 0x200000bd1:0xc39:0x0 500 /mnt/lustre/.lustre/.ltrash/MDT0000/0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd0:0x657:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbcc:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbd2:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbec:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc06:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc0c:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc12:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc1c:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc1f:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc25:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc28:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc2e:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc33:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc37:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc39:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/500: /mnt/lustre/.lustre/.ltrash/MDT0001: 0x240000404:0x5af:0x0 0x240000404:0x5b1:0x0 0x240000405:0x41:0x0 0x240000405:0xc:0x0 /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000404:0x5af:0x0: /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000404:0x5b1:0x0: file.1 /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000405:0x41:0x0: /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000405:0xc:0x0: lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x200000bd0:0x65f:0x0] 1 [0x240000404:0x5b1:0x0] mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117s (7s) == sanityn test 117t: UID/GID/ProjID for files in Trash Can ========================================================== 23:35:02 (1779075302) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain file: /mnt/lustre/d117t.sanityn/.Trash/tfile1, flags: 0X775B12DB00080002 unrm, uid: 0 gid: 0 projid: 0 timestamp: 1779075305 File: /mnt/lustre/d117t.sanityn/.Trash/tfile1 Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826937494 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-17 23:35:05.000000000 -0400 Modify: 2026-05-17 23:35:05.000000000 -0400 Change: 2026-05-17 23:35:05.000000000 -0400 Birth: 2026-05-17 23:35:05.000000000 -0400 trash_can_uid=4294967294 trash_can_gid=4294967294 trash_can_projid=4294967294 === Checking ugp ID on MDT === oleg135-server: debugfs 1.47.3-wc2 (11-Nov-2025) Inode: 519276 Type: regular Mode: 0644 Flags: 0x80002 Generation: 1009211183 Version: 0x00000002:00019659 User: -2 Group: -2 Project: -2 Size: 0 File ACL: 0 Links: 1 Blockcount: 0 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x6a0a88e9:00000000 -- Sun May 17 23:35:05 2026 atime: 0x6a0a88e9:00000000 -- Sun May 17 23:35:05 2026 mtime: 0x6a0a88e9:00000000 -- Sun May 17 23:35:05 2026 crtime: 0x6a0a88e9:6c3df600 -- Sun May 17 23:35:05 2026 Size of extra inode fields: 32 Extended attributes: lma: fid=[0x200000bd1:0xc96:0x0] compat=0 incompat=0 user.job (5) = "lfs.0" linkea: idx=0 parent=[0x200000bd0:0x663:0x0] name='tfile1' trusted.lov (56) = d0 0b d1 0b 01 00 00 00 96 0c 00 00 00 00 00 00 d1 0b 00 00 02 00 00 00 00 00 40 00 01 00 00 00 01 04 00 80 02 00 00 00 41 18 00 00 00 00 00 00 00 00 00 00 00 00 00 00 trusted.som (24) = 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 trusted.unrm (24) = 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 e9 88 0a 6a 00 00 00 00 EXTENTS: oleg135-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg135-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg135-server: debugfs 1.47.3-wc2 (11-Nov-2025) === Checking ugp ID on OST === oleg135-server: debugfs 1.47.3-wc2 (11-Nov-2025) Inode: 3484 Type: regular Mode: 0666 Flags: 0x80000 Generation: 3343147650 Version: 0x00000000:00000000 User: -2 Group: -2 Project: -2 Size: 0 File ACL: 0 Links: 1 Blockcount: 0 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x00000000:00000000 -- Wed Dec 31 19:00:00 1969 atime: 0x00000000:00000000 -- Wed Dec 31 19:00:00 1969 mtime: 0x00000000:00000000 -- Wed Dec 31 19:00:00 1969 crtime: 0x6a0a88ac:487ab000 -- Sun May 17 23:34:04 2026 Size of extra inode fields: 32 Extended attributes: lma: fid=[0x280000401:0x1841:0x0] compat=8 incompat=0 EXTENTS: oleg135-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg135-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg135-server: debugfs 1.47.3-wc2 (11-Nov-2025) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d117t.sanityn/tfile2] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [mkdir] [/mnt/lustre/d117t.sanityn/tdir2] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [unlink] [/mnt/lustre/d117t.sanityn/tfile2] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [rmdir] [/mnt/lustre/d117t.sanityn/tdir2] file: /mnt/lustre/d117t.sanityn/.Trash/tfile2, flags: 0X64ACD73700080002 unrm, uid: 500 gid: 500 projid: 0 timestamp: 1779075311 File: /mnt/lustre/d117t.sanityn/.Trash/tfile2 Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826937495 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 500/sanityusr) Gid: ( 500/sanityusr) Access: 2026-05-17 23:35:11.000000000 -0400 Modify: 2026-05-17 23:35:11.000000000 -0400 Change: 2026-05-17 23:35:11.000000000 -0400 Birth: 2026-05-17 23:35:11.000000000 -0400 using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg135-client.virtnet /mnt/lustre (opts:) Stopping client oleg135-client.virtnet /mnt/lustre opts: Stopping clients: oleg135-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg135-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg135-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg135-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg135-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg135-server tune2fs 1.47.3-wc2 (11-Nov-2025) tune2fs 1.47.3-wc2 (11-Nov-2025) tune2fs 1.47.3-wc2 (11-Nov-2025) tune2fs 1.47.3-wc2 (11-Nov-2025) sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) debugfs on /sys/kernel/debug type debugfs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg135-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.35,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg135-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.35,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg135-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.35,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/63884/testresults/sanityn-ldiskfs-DNE-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.35,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg135-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg135-server: oleg135-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-MDT0000 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg135-server: oleg135-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-MDT0001 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg135-server: oleg135-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg135-server: oleg135-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Started clients oleg135-client.virtnet: 192.168.201.135@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 Starting client oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 Started clients oleg135-client.virtnet: 192.168.201.135@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffffa0e720790800.idle_timeout=debug osc.lustre-OST0000-osc-ffffa0e72d09d800.idle_timeout=debug osc.lustre-OST0001-osc-ffffa0e720790800.idle_timeout=debug osc.lustre-OST0001-osc-ffffa0e72d09d800.idle_timeout=debug disable quota as required mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117t (112s) == sanityn test 117u: statfs() call should not account the space in Trash Can ========================================================== 23:36:54 (1779075414) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain mdd.lustre-MDT0000.trash_can_projid=1000 mdd.lustre-MDT0001.trash_can_projid=1000 llite.lustre-ffffa0e720790800.trash_can_projid=1000 llite.lustre-ffffa0e72d09d800.trash_can_projid=1000 total=7666232 used=85368 avail=7128672 free=7580864 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.135@tcp:/lustre 7666232 85368 7128672 2% /mnt/lustre 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 0.992746 s, 106 MB/s new total=7666232 used=85368 avail=7128672 free=7580864 Filesystem 1024-blocks Used Available Capacity Mounted on 192.168.201.135@tcp:/lustre 7666232 85368 7128672 2% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 5020 1266284 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 5636 1265668 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 154108 3452912 5% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 33664 3573356 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 187772 7026268 3% /mnt/lustre /mnt/lustre/.lustre/.ltrash: MDT0000 MDT0001 /mnt/lustre/.lustre/.ltrash/MDT0000: 0 0x200000bd0:0x657:0x0 0x200000bd1:0xbcc:0x0 0x200000bd1:0xbd2:0x0 0x200000bd1:0xbe3:0x0 0x200000bd1:0xbe7:0x0 0x200000bd1:0xbec:0x0 0x200000bd1:0xbf3:0x0 0x200000bd1:0xc06:0x0 0x200000bd1:0xc0c:0x0 0x200000bd1:0xc12:0x0 0x200000bd1:0xc1c:0x0 0x200000bd1:0xc1f:0x0 0x200000bd1:0xc25:0x0 0x200000bd1:0xc28:0x0 0x200000bd1:0xc2e:0x0 0x200000bd1:0xc33:0x0 0x200000bd1:0xc37:0x0 0x200000bd1:0xc39:0x0 0x200000bd1:0xc93:0x0 0x200000bd1:0xc95:0x0 0x2000013a1:0x3:0x0 500 /mnt/lustre/.lustre/.ltrash/MDT0000/0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd0:0x657:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbcc:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbd2:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbec:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc06:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc0c:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc12:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc1c:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc1f:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc25:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc28:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc2e:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc33:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc37:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc39:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc93:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc95:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3:0x0: f117u.sanityn /mnt/lustre/.lustre/.ltrash/MDT0000/500: /mnt/lustre/.lustre/.ltrash/MDT0001: 0x240000404:0x5af:0x0 0x240000404:0x5b1:0x0 0x240000405:0x41:0x0 0x240000405:0xc:0x0 /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000404:0x5af:0x0: /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000404:0x5b1:0x0: /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000405:0x41:0x0: /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000405:0xc:0x0: PROJID: 1000 Quota Info: Disk quotas for prj (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 102404 0 0 - 1 0 0 - lustre-MDT0000 0 - 0 - 1 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 102404 - 0 - 1 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Disk quotas for usr green (uid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 total=7666232 used=187772 avail=7026268 free=7478460 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 5020 1266284 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 5636 1265668 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 154108 3452912 5% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 33664 3573356 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 187772 7026268 3% /mnt/lustre new total=7666232 used=85368 avail=7128672 free=7580864 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 5020 1266284 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 5636 1265668 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 51704 3555316 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 33664 3573356 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 85368 7128672 2% /mnt/lustre llite.lustre-ffffa0e720790800.trash_can_projid=4294967294 llite.lustre-ffffa0e72d09d800.trash_can_projid=4294967294 mdd.lustre-MDT0000.trash_can_uid=4294967294 mdd.lustre-MDT0001.trash_can_uid=4294967294 mdd.lustre-MDT0000.trash_can_gid=4294967294 mdd.lustre-MDT0001.trash_can_gid=4294967294 mdd.lustre-MDT0000.trash_can_projid=4294967294 mdd.lustre-MDT0001.trash_can_projid=4294967294 mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117u (13s) == sanityn test 117ub: statfs() account the inode usage in Trash Can ========================================================== 23:37:07 (1779075427) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1024000 12539 1011461 2% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1024000 9321 1014679 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 262144 3681 258463 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 262144 3680 258464 2% /mnt/lustre[OST:1] filesystem_summary: 538787 21860 516927 5% /mnt/lustre total: 1000 open/close in 3.84 seconds: 260.73 ops/second - unlinked 0 (time 1779075437 ; total 0 ; last 0) total: 1000 unlinks in 2 seconds: 500.000000 unlinks/second === w/o Trash Can account === lustre-MDT0000_UUID 1024000 13541 1010459 2% /mnt/lustre[MDT:0] === w/ Trash Can account === lustre-MDT0000_UUID 1024000 12541 1011459 2% /mnt/lustre[MDT:0] tcu_ffree: 1011459 ffree: 1010459 mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117ub (24s) == sanityn test 117v: Test for Trash Can UID/GID/PROJID setting ========================================================== 23:37:32 (1779075452) mdd.lustre-MDT0000.trash_can_uid=10 mdd.lustre-MDT0001.trash_can_uid=10 mdd.lustre-MDT0000.trash_can_gid=10 mdd.lustre-MDT0001.trash_can_gid=10 mdd.lustre-MDT0000.trash_can_projid=10 mdd.lustre-MDT0001.trash_can_projid=10 llite.lustre-ffffa0e720790800.trash_can_projid=10 llite.lustre-ffffa0e72d09d800.trash_can_projid=10 mdd.lustre-MDT0000.trash_can_uid=-1 mdd.lustre-MDT0001.trash_can_uid=-1 mdd.lustre-MDT0000.trash_can_gid=-1 mdd.lustre-MDT0001.trash_can_gid=-1 mdd.lustre-MDT0000.trash_can_projid=-1 mdd.lustre-MDT0001.trash_can_projid=-1 llite.lustre-ffffa0e720790800.trash_can_projid=-1 llite.lustre-ffffa0e72d09d800.trash_can_projid=-1 mdd.lustre-MDT0000.trash_can_uid=-2 mdd.lustre-MDT0001.trash_can_uid=-2 mdd.lustre-MDT0000.trash_can_gid=-2 mdd.lustre-MDT0001.trash_can_gid=-2 mdd.lustre-MDT0000.trash_can_projid=-2 mdd.lustre-MDT0001.trash_can_projid=-2 llite.lustre-ffffa0e720790800.trash_can_projid=-2 llite.lustre-ffffa0e72d09d800.trash_can_projid=-2 mdd.lustre-MDT0000.trash_can_uid=original mdd.lustre-MDT0001.trash_can_uid=original mdd.lustre-MDT0000.trash_can_gid=original mdd.lustre-MDT0001.trash_can_gid=original mdd.lustre-MDT0000.trash_can_projid=original mdd.lustre-MDT0001.trash_can_projid=original llite.lustre-ffffa0e720790800.trash_can_projid=original llite.lustre-ffffa0e72d09d800.trash_can_projid=original mdd.lustre-MDT0000.trash_can_uid=default mdd.lustre-MDT0001.trash_can_uid=default mdd.lustre-MDT0000.trash_can_gid=default mdd.lustre-MDT0001.trash_can_gid=default mdd.lustre-MDT0000.trash_can_projid=default mdd.lustre-MDT0001.trash_can_projid=default llite.lustre-ffffa0e720790800.trash_can_projid=default llite.lustre-ffffa0e72d09d800.trash_can_projid=default llite.lustre-ffffa0e720790800.trash_can_projid=4294967294 llite.lustre-ffffa0e72d09d800.trash_can_projid=4294967294 mdd.lustre-MDT0000.trash_can_uid=4294967294 mdd.lustre-MDT0001.trash_can_uid=4294967294 mdd.lustre-MDT0000.trash_can_gid=4294967294 mdd.lustre-MDT0001.trash_can_gid=4294967294 mdd.lustre-MDT0000.trash_can_projid=4294967294 mdd.lustre-MDT0001.trash_can_projid=4294967294 PASS 117v (10s) == sanityn test 117w: Test Trash Can with original UID/GID/PROJID ========================================================== 23:37:42 (1779075462) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain mdd.lustre-MDT0000.trash_can_uid=original mdd.lustre-MDT0001.trash_can_uid=original mdd.lustre-MDT0000.trash_can_gid=original mdd.lustre-MDT0001.trash_can_gid=original mdd.lustre-MDT0000.trash_can_projid=original mdd.lustre-MDT0001.trash_can_projid=original 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.07404 s, 97.6 MB/s /mnt/lustre/.lustre/.ltrash: MDT0000 MDT0001 /mnt/lustre/.lustre/.ltrash/MDT0000: 0 0x200000bd0:0x657:0x0 0x200000bd1:0xbcc:0x0 0x200000bd1:0xbd2:0x0 0x200000bd1:0xbe3:0x0 0x200000bd1:0xbe7:0x0 0x200000bd1:0xbec:0x0 0x200000bd1:0xbf3:0x0 0x200000bd1:0xc06:0x0 0x200000bd1:0xc0c:0x0 0x200000bd1:0xc12:0x0 0x200000bd1:0xc1c:0x0 0x200000bd1:0xc1f:0x0 0x200000bd1:0xc25:0x0 0x200000bd1:0xc28:0x0 0x200000bd1:0xc2e:0x0 0x200000bd1:0xc33:0x0 0x200000bd1:0xc37:0x0 0x200000bd1:0xc39:0x0 0x200000bd1:0xc93:0x0 0x200000bd1:0xc95:0x0 0x2000013a1:0x3:0x0 0x2000013a1:0x3ee:0x0 0x2000013a1:0x5:0x0 500 /mnt/lustre/.lustre/.ltrash/MDT0000/0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd0:0x657:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbcc:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbd2:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbec:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc06:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc0c:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc12:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc1c:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc1f:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc25:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc28:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc2e:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc33:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc37:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc39:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc93:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc95:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3ee:0x0: tdir.1 tfile.1 /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3ee:0x0/tdir.1: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x5:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/500: /mnt/lustre/.lustre/.ltrash/MDT0001: 0x240000404:0x5af:0x0 0x240000404:0x5b1:0x0 0x240000405:0x41:0x0 0x240000405:0xc:0x0 /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000404:0x5af:0x0: /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000404:0x5b1:0x0: /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000405:0x41:0x0: /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000405:0xc:0x0: oleg135-server: debugfs 1.47.3-wc2 (11-Nov-2025) Inode: 519283 Type: regular Mode: 0644 Flags: 0x80000 Generation: 2196207117 Version: 0x00000003:000010bb User: 0 Group: 0 Project: 0 Size: 0 File ACL: 0 Links: 1 Blockcount: 0 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x6a0a898b:00000000 -- Sun May 17 23:37:47 2026 atime: 0x6a0a898a:00000000 -- Sun May 17 23:37:46 2026 mtime: 0x6a0a898b:00000000 -- Sun May 17 23:37:47 2026 crtime: 0x6a0a898a:7b432d00 -- Sun May 17 23:37:46 2026 Size of extra inode fields: 32 Extended attributes: lma: fid=[0x2000013a1:0x3f0:0x0] compat=0 incompat=0 trusted.lov (56) = d0 0b d1 0b 01 00 00 00 f0 03 00 00 00 00 00 00 a1 13 00 00 02 00 00 00 00 00 40 00 01 00 00 00 01 04 00 c0 02 00 00 00 57 1a 00 00 00 00 00 00 00 00 00 00 01 00 00 00 user.job (4) = "dd.0" linkea: idx=0 parent=[0x2000013a0:0x3:0x0] name='tfile.1' trusted.som (24) = 04 00 00 00 00 00 00 00 00 00 40 06 00 00 00 00 08 20 03 00 00 00 00 00 EXTENTS: oleg135-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg135-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg135-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg135-server: debugfs 1.47.3-wc2 (11-Nov-2025) Inode: 519282 Type: directory Mode: 0755 Flags: 0x0 Generation: 2342977034 Version: 0x00000003:000010ba User: 0 Group: 0 Project: 0 Size: 4096 File ACL: 0 Links: 4 Blockcount: 8 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x6a0a898c:00000000 -- Sun May 17 23:37:48 2026 atime: 0x6a0a898a:00000000 -- Sun May 17 23:37:46 2026 mtime: 0x6a0a898a:00000000 -- Sun May 17 23:37:46 2026 crtime: 0x6a0a898a:78e0d300 -- Sun May 17 23:37:46 2026 Size of extra inode fields: 32 Extended attributes: lma: fid=[0x2000013a1:0x3ef:0x0] compat=0 incompat=0 user.job (7) = "mkdir.0" linkea: idx=0 parent=[0x2000013a0:0x3:0x0] name='tdir.1' BLOCKS: (0):345714 TOTAL: 1 oleg135-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg135-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg135-server: debugfs 1.47.3-wc2 (11-Nov-2025) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [mkdir] [/mnt/lustre/d117w.sanityn/tdir.2] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d117w.sanityn/tfile.2] oleg135-server: debugfs 1.47.3-wc2 (11-Nov-2025) Inode: 519286 Type: regular Mode: 0644 Flags: 0x80000 Generation: 1100231688 Version: 0x00000003:000010f3 User: 500 Group: 500 Project: 0 Size: 0 File ACL: 0 Links: 1 Blockcount: 0 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x6a0a898f:00000000 -- Sun May 17 23:37:51 2026 atime: 0x6a0a898f:00000000 -- Sun May 17 23:37:51 2026 mtime: 0x6a0a898f:00000000 -- Sun May 17 23:37:51 2026 crtime: 0x6a0a898e:d9330700 -- Sun May 17 23:37:50 2026 Size of extra inode fields: 32 Extended attributes: lma: fid=[0x2000013a1:0x3f2:0x0] compat=0 incompat=0 trusted.lov (56) = d0 0b d1 0b 01 00 00 00 f2 03 00 00 00 00 00 00 a1 13 00 00 02 00 00 00 00 00 40 00 01 00 00 00 01 04 00 80 02 00 00 00 57 1a 00 00 00 00 00 00 00 00 00 00 00 00 00 00 user.job (9) = "touch.500" linkea: idx=0 parent=[0x2000013a0:0x3:0x0] name='tfile.2' trusted.som (24) = 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 EXTENTS: oleg135-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg135-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg135-server: debugfs 1.47.3-wc2 (11-Nov-2025) Inode: 519285 Type: directory Mode: 0755 Flags: 0x0 Generation: 2881710356 Version: 0x00000003:000010f2 User: 500 Group: 500 Project: 0 Size: 4096 File ACL: 0 Links: 4 Blockcount: 8 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x6a0a898f:00000000 -- Sun May 17 23:37:51 2026 atime: 0x6a0a898f:00000000 -- Sun May 17 23:37:51 2026 mtime: 0x6a0a898f:00000000 -- Sun May 17 23:37:51 2026 crtime: 0x6a0a898e:d6d0ad00 -- Sun May 17 23:37:50 2026 Size of extra inode fields: 32 Extended attributes: lma: fid=[0x2000013a1:0x3f1:0x0] compat=0 incompat=0 user.job (9) = "mkdir.500" linkea: idx=0 parent=[0x2000013a0:0x3:0x0] name='tdir.2' BLOCKS: (0):345716 TOTAL: 1 oleg135-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg135-server: debugfs 1.47.3-wc2 (11-Nov-2025) mdd.lustre-MDT0000.trash_can_uid=4294967294 mdd.lustre-MDT0001.trash_can_uid=4294967294 mdd.lustre-MDT0000.trash_can_gid=4294967294 mdd.lustre-MDT0001.trash_can_gid=4294967294 mdd.lustre-MDT0000.trash_can_projid=4294967294 mdd.lustre-MDT0001.trash_can_projid=4294967294 mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117w (15s) == sanityn test 117x: Unrm Files from Trash Can should also restore UID/GID/PROJID ========================================================== 23:37:57 (1779075477) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/d117x.sanityn/.Trash/tdir1 trusted.unrm=0sAAAAAAAAAAAAAAAAAAAAAJeJCmoAAAAA getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/d117x.sanityn/.Trash/tfile1 trusted.unrm=0sAAAAAAAAAAAAAAAAAAAAAJeJCmoAAAAA /mnt/lustre/d117x.sanityn/tdir1: trusted.unrm: No such attribute /mnt/lustre/d117x.sanityn/tfile1: trusted.unrm: No such attribute running as uid/gid/euid/egid 500/500/500/500, groups: 500 [mkdir] [/mnt/lustre/d117x.sanityn/tdir2] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d117x.sanityn/tfile2] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [rmdir] [/mnt/lustre/d117x.sanityn/tdir2] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [unlink] [/mnt/lustre/d117x.sanityn/tfile2] getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/d117x.sanityn/.Trash/tdir2 trusted.unrm=0sAAAAAPQBAAD0AQAAAAAAAJeJCmoAAAAA getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/d117x.sanityn/.Trash/tfile2 trusted.unrm=0sAAAAAPQBAAD0AQAAAAAAAJeJCmoAAAAA /mnt/lustre/d117x.sanityn/tdir2: trusted.unrm: No such attribute /mnt/lustre/d117x.sanityn/tfile2: trusted.unrm: No such attribute mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117x (6s) == sanityn test 117ya: Test with symbol links with Trash Can ========================================================== 23:38:03 (1779075483) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain total 8 lrwxrwxrwx 1 root root 32 May 17 23:38 lnkd5 -> /mnt/lustre/d117ya.sanityn/tdir1 lrwxrwxrwx 1 root root 33 May 17 23:38 lnkf4 -> /mnt/lustre/d117ya.sanityn/tfile3 drwxr-xr-x 3 root root 4096 May 17 23:38 tdir1 drwxr-xr-x 2 root root 4096 May 17 23:38 tdir2 -rw-r--r-- 1 root root 0 May 17 23:38 tfile3 After unlink /mnt/lustre/d117ya.sanityn/lnkf4 and /mnt/lustre/d117ya.sanityn/lnkd5 /mnt/lustre/d117ya.sanityn/.Trash: total 4 lrwxrwxrwx 1 root root 32 May 17 23:38 lnkd5 -> /mnt/lustre/d117ya.sanityn/tdir1 lrwxrwxrwx 1 root root 33 May 17 23:38 lnkf4 -> /mnt/lustre/d117ya.sanityn/tfile3 drwxr-xr-x 3 root root 4096 May 17 23:38 tdir1 -rw-r--r-- 1 root root 0 May 17 23:38 tfile3 /mnt/lustre/d117ya.sanityn/.Trash/tdir1: total 4 drwxr-xr-x 4 root root 4096 May 17 23:38 tdir13 -rw-r--r-- 1 root root 0 May 17 23:38 tfile11 -rw-r--r-- 1 root root 0 May 17 23:38 tfile12 /mnt/lustre/d117ya.sanityn/.Trash/tdir1/tdir13: total 0 file: /mnt/lustre/d117ya.sanityn/.Trash/lnkf4, flags: 0X2 unrm, uid: 0 gid: 0 projid: 0 timestamp: 1779075486 file: /mnt/lustre/d117ya.sanityn/.Trash/lnkd5, flags: 0X2 unrm, uid: 0 gid: 0 projid: 0 timestamp: 1779075486 Unrm stub file 'lnkd5' under directory '/mnt/lustre/d117ya.sanityn' Check if '/mnt/lustre/d117ya.sanityn/.Trash/lnkd5' is in Trash Can Path '/mnt/lustre/d117ya.sanityn/.Trash/lnkd5' is a symlink Handle symlink /mnt/lustre/d117ya.sanityn/.Trash/lnkd5 Unrm stub file 'lnkf4' under directory '/mnt/lustre/d117ya.sanityn' Check if '/mnt/lustre/d117ya.sanityn/.Trash/lnkf4' is in Trash Can Path '/mnt/lustre/d117ya.sanityn/.Trash/lnkf4' is a symlink Handle symlink /mnt/lustre/d117ya.sanityn/.Trash/lnkf4 Unrm stub file 'tfile3' under directory '/mnt/lustre/d117ya.sanityn' Check if '/mnt/lustre/d117ya.sanityn/.Trash/tfile3' is in Trash Can Set flags 0X80002/0X80000 OK: rc=0 Unrm stub file 'tdir1' under directory '/mnt/lustre/d117ya.sanityn' Check if '/mnt/lustre/d117ya.sanityn/.Trash/tdir1' is in Trash Can Set flags 0X2/0 OK: rc=0 Handle file 0:'tfile11' under directory '/mnt/lustre/d117ya.sanityn/tdir1' Handle special file 0:/mnt/lustre/d117ya.sanityn/tdir1/tfile11 Handle file 0:'tfile12' under directory '/mnt/lustre/d117ya.sanityn/tdir1' Handle special file 0:/mnt/lustre/d117ya.sanityn/tdir1/tfile12 Handle dir 0:'tdir13' under directory '/mnt/lustre/d117ya.sanityn/tdir1' Handle special file 0:/mnt/lustre/d117ya.sanityn/tdir1/tdir13 mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117ya (7s) == sanityn test 117yb: Compare the directories containing symlinks after unrm recover ========================================================== 23:38:10 (1779075490) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain Only in /etc/pam.d: password-auth Only in /etc/postfix: master.cf Only in /etc: subuid Only in /etc: subuid- sanityn test_117yb: @@@@@@ FAIL: /etc and /mnt/lustre/d117yb.sanityn/etc diff Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7318:error() = /home/green/git/lustre-release/lustre/tests/sanityn.sh:8920:test_117yb() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7694:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7757:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7572:run_test() = /home/green/git/lustre-release/lustre/tests/sanityn.sh:8922:main() Dumping lctl log to /tmp/testlogs//sanityn.test_117yb.*.1779075534.log rsync: chown "/tmp/testlogs/.sanityn.test_117yb.debug_log.oleg135-server.1779075534.log.Aaulq7" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanityn.test_117yb.dmesg.oleg135-server.1779075534.log.ygNrnZ" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid FAIL 117yb (46s) == sanityn test 117yc: Restoring UID/GID/PROJID for symlinks works ========================================================== 23:38:56 (1779075536) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg135-client.virtnet /mnt/lustre (opts:) Stopping client oleg135-client.virtnet /mnt/lustre opts: Stopping clients: oleg135-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg135-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg135-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg135-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg135-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg135-server tune2fs 1.47.3-wc2 (11-Nov-2025) tune2fs 1.47.3-wc2 (11-Nov-2025) tune2fs 1.47.3-wc2 (11-Nov-2025) tune2fs 1.47.3-wc2 (11-Nov-2025) sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) debugfs on /sys/kernel/debug type debugfs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg135-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.35,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg135-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.35,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg135-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.35,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/63884/testresults/sanityn-ldiskfs-DNE-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.35,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg135-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg135-server: oleg135-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-MDT0000 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg135-server: oleg135-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-MDT0001 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg135-server: oleg135-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg135-server: oleg135-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Started clients oleg135-client.virtnet: 192.168.201.135@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 Starting client oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 Started clients oleg135-client.virtnet: 192.168.201.135@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffffa0e704f1e000.idle_timeout=debug osc.lustre-OST0000-osc-ffffa0e7050cf000.idle_timeout=debug osc.lustre-OST0001-osc-ffffa0e704f1e000.idle_timeout=debug osc.lustre-OST0001-osc-ffffa0e7050cf000.idle_timeout=debug disable quota as required mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain running as uid/gid/euid/egid 500/500/500/500, groups: 500 [mkdir] [/mnt/lustre/d117yc.sanityn/tdir1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d117yc.sanityn/tfile1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [ln] [-s] [/mnt/lustre/d117yc.sanityn/tfile1] [/mnt/lustre/d117yc.sanityn/lnkf3] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [ln] [-s] [/mnt/lustre/d117yc.sanityn/tdir1] [/mnt/lustre/d117yc.sanityn/lnkd4] getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/d117yc.sanityn/.Trash/lnkf3 trusted.unrm=0sAAAAAPQBAAD0AQAAyAAAACGKCmoAAAAA getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/d117yc.sanityn/.Trash/lnkd4 trusted.unrm=0sAAAAAPQBAAD0AQAAAAAAACGKCmoAAAAA Unrm stub file 'tfile1' under directory '/mnt/lustre/d117yc.sanityn' Check if '/mnt/lustre/d117yc.sanityn/.Trash/tfile1' is in Trash Can Set flags 0X20080002/0X20080000 OK: rc=0 Unrm stub file 'lnkf3' under directory '/mnt/lustre/d117yc.sanityn' Check if '/mnt/lustre/d117yc.sanityn/.Trash/lnkf3' is in Trash Can Path '/mnt/lustre/d117yc.sanityn/.Trash/lnkf3' is a symlink Handle symlink /mnt/lustre/d117yc.sanityn/.Trash/lnkf3 Unrm stub file 'lnkd4' under directory '/mnt/lustre/d117yc.sanityn' Check if '/mnt/lustre/d117yc.sanityn/.Trash/lnkd4' is in Trash Can Path '/mnt/lustre/d117yc.sanityn/.Trash/lnkd4' is a symlink Handle symlink /mnt/lustre/d117yc.sanityn/.Trash/lnkd4 Unrm stub file 'tdir1' under directory '/mnt/lustre/d117yc.sanityn' Check if '/mnt/lustre/d117yc.sanityn/.Trash/tdir1' is in Trash Can Set flags 0X2/0 OK: rc=0 /mnt/lustre/d117yc.sanityn/lnkf3: trusted.unrm: No such attribute /mnt/lustre/d117yc.sanityn/lnkd4: trusted.unrm: No such attribute mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117yc (85s) == sanityn test 117za: Test Trash Can for UID Trash Can type with nodemap ========================================================== 23:40:21 (1779075621) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid Trash Can Type: uid /mnt/lustre/.lustre/.ltrash: MDT0000 MDT0001 /mnt/lustre/.lustre/.ltrash/MDT0000: 0 0x200000bd0:0x657:0x0 0x200000bd1:0xbcc:0x0 0x200000bd1:0xbd2:0x0 0x200000bd1:0xbe3:0x0 0x200000bd1:0xbe7:0x0 0x200000bd1:0xbec:0x0 0x200000bd1:0xbf3:0x0 0x200000bd1:0xc06:0x0 0x200000bd1:0xc0c:0x0 0x200000bd1:0xc12:0x0 0x200000bd1:0xc1c:0x0 0x200000bd1:0xc1f:0x0 0x200000bd1:0xc25:0x0 0x200000bd1:0xc28:0x0 0x200000bd1:0xc2e:0x0 0x200000bd1:0xc33:0x0 0x200000bd1:0xc37:0x0 0x200000bd1:0xc39:0x0 0x200000bd1:0xc93:0x0 0x200000bd1:0xc95:0x0 0x2000013a1:0x3:0x0 0x2000013a1:0x3ee:0x0 0x2000013a1:0x3f3:0x0 0x2000013a1:0x3f8:0x0 0x2000013a1:0x401:0x0 0x2000013a1:0x5:0x0 0x200001b71:0x1:0x0 500 /mnt/lustre/.lustre/.ltrash/MDT0000/0: 0x200001b71:0x6:0x0 /mnt/lustre/.lustre/.ltrash/MDT0000/0/0x200001b71:0x6:0x0: tfile.0 /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd0:0x657:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbcc:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbd2:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbec:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc06:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc0c:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc12:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc1c:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc1f:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc25:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc28:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc2e:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc33:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc37:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc39:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc93:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc95:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3ee:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3f3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3f8:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x401:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x5:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0x1:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/500: /mnt/lustre/.lustre/.ltrash/MDT0001: 0x240000404:0x5af:0x0 0x240000404:0x5b1:0x0 0x240000405:0x41:0x0 0x240000405:0xc:0x0 /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000404:0x5af:0x0: /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000404:0x5b1:0x0: /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000405:0x41:0x0: /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000405:0xc:0x0: On MGS 192.168.201.135, active = nodemap.active=1 waiting 10 secs for sync On MGS 192.168.201.135, tnm.trusted_nodemap = nodemap.tnm.trusted_nodemap=1 waiting 10 secs for sync nodemap.tnm.idmap= [ { idtype: uid, client_id: 500, fs_id: 10000 }, { idtype: uid, client_id: 501, fs_id: 10001 }, { idtype: uid, client_id: 502, fs_id: 10002 }, { idtype: uid, client_id: 503, fs_id: 10003 }, { idtype: uid, client_id: 504, fs_id: 10004 }, { idtype: uid, client_id: 505, fs_id: 10005 }, { idtype: uid, client_id: 506, fs_id: 10006 }, { idtype: uid, client_id: 507, fs_id: 10007 }, { idtype: uid, client_id: 508, fs_id: 10008 }, { idtype: uid, client_id: 509, fs_id: 10009 } ] /mnt/lustre/.lustre/.ltrash: MDT0000 MDT0001 /mnt/lustre/.lustre/.ltrash/MDT0000: 0 0x200000bd0:0x657:0x0 0x200000bd1:0xbcc:0x0 0x200000bd1:0xbd2:0x0 0x200000bd1:0xbe3:0x0 0x200000bd1:0xbe7:0x0 0x200000bd1:0xbec:0x0 0x200000bd1:0xbf3:0x0 0x200000bd1:0xc06:0x0 0x200000bd1:0xc0c:0x0 0x200000bd1:0xc12:0x0 0x200000bd1:0xc1c:0x0 0x200000bd1:0xc1f:0x0 0x200000bd1:0xc25:0x0 0x200000bd1:0xc28:0x0 0x200000bd1:0xc2e:0x0 0x200000bd1:0xc33:0x0 0x200000bd1:0xc37:0x0 0x200000bd1:0xc39:0x0 0x200000bd1:0xc93:0x0 0x200000bd1:0xc95:0x0 0x2000013a1:0x3:0x0 0x2000013a1:0x3ee:0x0 0x2000013a1:0x3f3:0x0 0x2000013a1:0x3f8:0x0 0x2000013a1:0x401:0x0 0x2000013a1:0x5:0x0 0x200001b71:0x1:0x0 500 tnm /mnt/lustre/.lustre/.ltrash/MDT0000/0: 0x200001b71:0x6:0x0 /mnt/lustre/.lustre/.ltrash/MDT0000/0/0x200001b71:0x6:0x0: tfile.0 /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd0:0x657:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbcc:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbd2:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbec:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc06:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc0c:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc12:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc1c:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc1f:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc25:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc28:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc2e:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc33:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc37:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc39:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc93:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc95:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3ee:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3f3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3f8:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x401:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x5:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0x1:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/500: /mnt/lustre/.lustre/.ltrash/MDT0000/tnm: 0 /mnt/lustre/.lustre/.ltrash/MDT0000/tnm/0: 0x200001b71:0x6:0x0 /mnt/lustre/.lustre/.ltrash/MDT0000/tnm/0/0x200001b71:0x6:0x0: tfile.1 /mnt/lustre/.lustre/.ltrash/MDT0001: 0x240000404:0x5af:0x0 0x240000404:0x5b1:0x0 0x240000405:0x41:0x0 0x240000405:0xc:0x0 /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000404:0x5af:0x0: /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000404:0x5b1:0x0: /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000405:0x41:0x0: /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000405:0xc:0x0: running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d117za.sanityn/tfile.2] /mnt/lustre/.lustre/.ltrash: MDT0000 MDT0001 /mnt/lustre/.lustre/.ltrash/MDT0000: 0 0x200000bd0:0x657:0x0 0x200000bd1:0xbcc:0x0 0x200000bd1:0xbd2:0x0 0x200000bd1:0xbe3:0x0 0x200000bd1:0xbe7:0x0 0x200000bd1:0xbec:0x0 0x200000bd1:0xbf3:0x0 0x200000bd1:0xc06:0x0 0x200000bd1:0xc0c:0x0 0x200000bd1:0xc12:0x0 0x200000bd1:0xc1c:0x0 0x200000bd1:0xc1f:0x0 0x200000bd1:0xc25:0x0 0x200000bd1:0xc28:0x0 0x200000bd1:0xc2e:0x0 0x200000bd1:0xc33:0x0 0x200000bd1:0xc37:0x0 0x200000bd1:0xc39:0x0 0x200000bd1:0xc93:0x0 0x200000bd1:0xc95:0x0 0x2000013a1:0x3:0x0 0x2000013a1:0x3ee:0x0 0x2000013a1:0x3f3:0x0 0x2000013a1:0x3f8:0x0 0x2000013a1:0x401:0x0 0x2000013a1:0x5:0x0 0x200001b71:0x1:0x0 500 tnm /mnt/lustre/.lustre/.ltrash/MDT0000/0: 0x200001b71:0x6:0x0 /mnt/lustre/.lustre/.ltrash/MDT0000/0/0x200001b71:0x6:0x0: tfile.0 /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd0:0x657:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbcc:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbd2:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbec:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc06:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc0c:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc12:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc1c:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc1f:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc25:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc28:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc2e:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc33:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc37:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc39:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc93:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc95:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3ee:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3f3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3f8:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x401:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x5:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0x1:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/500: /mnt/lustre/.lustre/.ltrash/MDT0000/tnm: 0 500 /mnt/lustre/.lustre/.ltrash/MDT0000/tnm/0: 0x200001b71:0x6:0x0 /mnt/lustre/.lustre/.ltrash/MDT0000/tnm/0/0x200001b71:0x6:0x0: tfile.1 /mnt/lustre/.lustre/.ltrash/MDT0000/tnm/500: 0x200001b71:0x6:0x0 /mnt/lustre/.lustre/.ltrash/MDT0000/tnm/500/0x200001b71:0x6:0x0: tfile.2 /mnt/lustre/.lustre/.ltrash/MDT0001: 0x240000404:0x5af:0x0 0x240000404:0x5b1:0x0 0x240000405:0x41:0x0 0x240000405:0xc:0x0 /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000404:0x5af:0x0: /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000404:0x5b1:0x0: /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000405:0x41:0x0: /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000405:0xc:0x0: oleg135-server: debugfs 1.47.3-wc2 (11-Nov-2025) Inode: 520185 Type: regular Mode: 0644 Flags: 0x80002 Generation: 3210168806 Version: 0x00000004:000000fd User: -2 Group: -2 Project: -2 Size: 0 File ACL: 0 Links: 1 Blockcount: 0 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x6a0a8a44:00000000 -- Sun May 17 23:40:52 2026 atime: 0x6a0a8a44:00000000 -- Sun May 17 23:40:52 2026 mtime: 0x6a0a8a44:00000000 -- Sun May 17 23:40:52 2026 crtime: 0x6a0a8a43:9e124f00 -- Sun May 17 23:40:51 2026 Size of extra inode fields: 32 Extended attributes: lma: fid=[0x200001b71:0x9:0x0] compat=0 incompat=0 trusted.lov (56) = d0 0b d1 0b 01 00 00 00 09 00 00 00 00 00 00 00 71 1b 00 00 02 00 00 00 00 00 40 00 01 00 00 00 01 04 00 80 02 00 00 00 63 1b 00 00 00 00 00 00 00 00 00 00 00 00 00 00 user.job (9) = "touch.500" linkea: idx=0 parent=[0x200001b70:0x7:0x0] name='tfile.2' trusted.som (24) = 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 trusted.unrm (24) = 00 00 00 00 f4 01 00 00 f4 01 00 00 00 00 00 00 43 8a 0a 6a 00 00 00 00 EXTENTS: file: /mnt/lustre/.lustre/.ltrash/MDT0000/tnm/500/0x200001b71:0x6:0x0/tfile.2, flags: 0XB0B903D300080002 unrm, uid: 500 gid: 500 projid: 0 timestamp: 1779075651 On MGS 192.168.201.135, active = nodemap.active=0 waiting 10 secs for sync mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117za (47s) == sanityn test 117zb: Test Trash Can for PLAIN Trash Can type with nodemap ========================================================== 23:41:09 (1779075669) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain /mnt/lustre/.lustre/.ltrash: MDT0000 MDT0001 /mnt/lustre/.lustre/.ltrash/MDT0000: 0 0x200000bd0:0x657:0x0 0x200000bd1:0xbcc:0x0 0x200000bd1:0xbd2:0x0 0x200000bd1:0xbe3:0x0 0x200000bd1:0xbe7:0x0 0x200000bd1:0xbec:0x0 0x200000bd1:0xbf3:0x0 0x200000bd1:0xc06:0x0 0x200000bd1:0xc0c:0x0 0x200000bd1:0xc12:0x0 0x200000bd1:0xc1c:0x0 0x200000bd1:0xc1f:0x0 0x200000bd1:0xc25:0x0 0x200000bd1:0xc28:0x0 0x200000bd1:0xc2e:0x0 0x200000bd1:0xc33:0x0 0x200000bd1:0xc37:0x0 0x200000bd1:0xc39:0x0 0x200000bd1:0xc93:0x0 0x200000bd1:0xc95:0x0 0x2000013a1:0x3:0x0 0x2000013a1:0x3ee:0x0 0x2000013a1:0x3f3:0x0 0x2000013a1:0x3f8:0x0 0x2000013a1:0x401:0x0 0x2000013a1:0x5:0x0 0x200001b71:0x1:0x0 0x200001b71:0xa:0x0 500 tnm /mnt/lustre/.lustre/.ltrash/MDT0000/0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd0:0x657:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbcc:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbd2:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbec:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc06:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc0c:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc12:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc1c:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc1f:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc25:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc28:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc2e:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc33:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc37:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc39:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc93:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc95:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3ee:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3f3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3f8:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x401:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x5:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0x1:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0xa:0x0: tfile.0 /mnt/lustre/.lustre/.ltrash/MDT0000/500: /mnt/lustre/.lustre/.ltrash/MDT0000/tnm: /mnt/lustre/.lustre/.ltrash/MDT0001: 0x240000404:0x5af:0x0 0x240000404:0x5b1:0x0 0x240000405:0x41:0x0 0x240000405:0xc:0x0 /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000404:0x5af:0x0: /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000404:0x5b1:0x0: /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000405:0x41:0x0: /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000405:0xc:0x0: On MGS 192.168.201.135, active = nodemap.active=1 waiting 10 secs for sync On MGS 192.168.201.135, tnm.trusted_nodemap = nodemap.tnm.trusted_nodemap=1 waiting 10 secs for sync nodemap.tnm.idmap= [ { idtype: uid, client_id: 500, fs_id: 10000 }, { idtype: uid, client_id: 501, fs_id: 10001 }, { idtype: uid, client_id: 502, fs_id: 10002 }, { idtype: uid, client_id: 503, fs_id: 10003 }, { idtype: uid, client_id: 504, fs_id: 10004 }, { idtype: uid, client_id: 505, fs_id: 10005 }, { idtype: uid, client_id: 506, fs_id: 10006 }, { idtype: uid, client_id: 507, fs_id: 10007 }, { idtype: uid, client_id: 508, fs_id: 10008 }, { idtype: uid, client_id: 509, fs_id: 10009 } ] /mnt/lustre/.lustre/.ltrash: MDT0000 MDT0001 /mnt/lustre/.lustre/.ltrash/MDT0000: 0 0x200000bd0:0x657:0x0 0x200000bd1:0xbcc:0x0 0x200000bd1:0xbd2:0x0 0x200000bd1:0xbe3:0x0 0x200000bd1:0xbe7:0x0 0x200000bd1:0xbec:0x0 0x200000bd1:0xbf3:0x0 0x200000bd1:0xc06:0x0 0x200000bd1:0xc0c:0x0 0x200000bd1:0xc12:0x0 0x200000bd1:0xc1c:0x0 0x200000bd1:0xc1f:0x0 0x200000bd1:0xc25:0x0 0x200000bd1:0xc28:0x0 0x200000bd1:0xc2e:0x0 0x200000bd1:0xc33:0x0 0x200000bd1:0xc37:0x0 0x200000bd1:0xc39:0x0 0x200000bd1:0xc93:0x0 0x200000bd1:0xc95:0x0 0x2000013a1:0x3:0x0 0x2000013a1:0x3ee:0x0 0x2000013a1:0x3f3:0x0 0x2000013a1:0x3f8:0x0 0x2000013a1:0x401:0x0 0x2000013a1:0x5:0x0 0x200001b71:0x1:0x0 0x200001b71:0xa:0x0 500 tnm /mnt/lustre/.lustre/.ltrash/MDT0000/0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd0:0x657:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbcc:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbd2:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbec:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc06:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc0c:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc12:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc1c:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc1f:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc25:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc28:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc2e:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc33:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc37:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc39:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc93:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc95:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3ee:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3f3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3f8:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x401:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x5:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0x1:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0xa:0x0: tfile.0 /mnt/lustre/.lustre/.ltrash/MDT0000/500: /mnt/lustre/.lustre/.ltrash/MDT0000/tnm: 0x200001b71:0xa:0x0 /mnt/lustre/.lustre/.ltrash/MDT0000/tnm/0x200001b71:0xa:0x0: tfile.1 /mnt/lustre/.lustre/.ltrash/MDT0001: 0x240000404:0x5af:0x0 0x240000404:0x5b1:0x0 0x240000405:0x41:0x0 0x240000405:0xc:0x0 /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000404:0x5af:0x0: /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000404:0x5b1:0x0: /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000405:0x41:0x0: /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000405:0xc:0x0: running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d117zb.sanityn/tfile.2] /mnt/lustre/.lustre/.ltrash: MDT0000 MDT0001 /mnt/lustre/.lustre/.ltrash/MDT0000: 0 0x200000bd0:0x657:0x0 0x200000bd1:0xbcc:0x0 0x200000bd1:0xbd2:0x0 0x200000bd1:0xbe3:0x0 0x200000bd1:0xbe7:0x0 0x200000bd1:0xbec:0x0 0x200000bd1:0xbf3:0x0 0x200000bd1:0xc06:0x0 0x200000bd1:0xc0c:0x0 0x200000bd1:0xc12:0x0 0x200000bd1:0xc1c:0x0 0x200000bd1:0xc1f:0x0 0x200000bd1:0xc25:0x0 0x200000bd1:0xc28:0x0 0x200000bd1:0xc2e:0x0 0x200000bd1:0xc33:0x0 0x200000bd1:0xc37:0x0 0x200000bd1:0xc39:0x0 0x200000bd1:0xc93:0x0 0x200000bd1:0xc95:0x0 0x2000013a1:0x3:0x0 0x2000013a1:0x3ee:0x0 0x2000013a1:0x3f3:0x0 0x2000013a1:0x3f8:0x0 0x2000013a1:0x401:0x0 0x2000013a1:0x5:0x0 0x200001b71:0x1:0x0 0x200001b71:0xa:0x0 500 tnm /mnt/lustre/.lustre/.ltrash/MDT0000/0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd0:0x657:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbcc:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbd2:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbec:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc06:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc0c:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc12:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc1c:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc1f:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc25:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc28:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc2e:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc33:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc37:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc39:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc93:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc95:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3ee:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3f3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3f8:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x401:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x5:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0x1:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0xa:0x0: tfile.0 /mnt/lustre/.lustre/.ltrash/MDT0000/500: /mnt/lustre/.lustre/.ltrash/MDT0000/tnm: 0x200001b71:0xa:0x0 /mnt/lustre/.lustre/.ltrash/MDT0000/tnm/0x200001b71:0xa:0x0: tfile.1 tfile.2 /mnt/lustre/.lustre/.ltrash/MDT0001: 0x240000404:0x5af:0x0 0x240000404:0x5b1:0x0 0x240000405:0x41:0x0 0x240000405:0xc:0x0 /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000404:0x5af:0x0: /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000404:0x5b1:0x0: /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000405:0x41:0x0: /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000405:0xc:0x0: oleg135-server: debugfs 1.47.3-wc2 (11-Nov-2025) Inode: 520185 Type: regular Mode: 0644 Flags: 0x80002 Generation: 389140109 Version: 0x00000004:0000021e User: -2 Group: -2 Project: -2 Size: 0 File ACL: 0 Links: 1 Blockcount: 0 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x6a0a8a70:00000000 -- Sun May 17 23:41:36 2026 atime: 0x6a0a8a70:00000000 -- Sun May 17 23:41:36 2026 mtime: 0x6a0a8a70:00000000 -- Sun May 17 23:41:36 2026 crtime: 0x6a0a8a70:3ccbf700 -- Sun May 17 23:41:36 2026 Size of extra inode fields: 32 Extended attributes: lma: fid=[0x200001b71:0xd:0x0] compat=0 incompat=0 trusted.lov (56) = d0 0b d1 0b 01 00 00 00 0d 00 00 00 00 00 00 00 71 1b 00 00 02 00 00 00 00 00 40 00 01 00 00 00 01 04 00 c0 02 00 00 00 65 1b 00 00 00 00 00 00 00 00 00 00 01 00 00 00 user.job (9) = "touch.500" linkea: idx=0 parent=[0x200001b70:0x9:0x0] name='tfile.2' trusted.som (24) = 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 trusted.unrm (24) = 00 00 00 00 f4 01 00 00 f4 01 00 00 00 00 00 00 70 8a 0a 6a 00 00 00 00 EXTENTS: file: /mnt/lustre/.lustre/.ltrash/MDT0000/tnm//0x200001b71:0xa:0x0/tfile.2, flags: 0XF415093E00080002 unrm, uid: 500 gid: 500 projid: 0 timestamp: 1779075696 On MGS 192.168.201.135, active = nodemap.active=0 waiting 10 secs for sync mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117zb (45s) == sanityn test 117zc: TCU: Test striped dir for UID Trash Can type ========================================================== 23:41:53 (1779075713) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid Trash Can Type: uid Creating 100 files under striped directory: /mnt/lustre/d117zc.sanityn/striped_dir total: 100 open/close in 0.50 seconds: 198.12 ops/second f117zc.sanityn0 -> 1 f117zc.sanityn1 -> 1 f117zc.sanityn2 -> 1 f117zc.sanityn3 -> 1 f117zc.sanityn4 -> 1 f117zc.sanityn5 -> 1 f117zc.sanityn6 -> 1 f117zc.sanityn7 -> 1 f117zc.sanityn8 -> 1 f117zc.sanityn9 -> 1 f117zc.sanityn10 -> 0 f117zc.sanityn11 -> 0 f117zc.sanityn12 -> 0 f117zc.sanityn13 -> 1 f117zc.sanityn14 -> 0 f117zc.sanityn15 -> 1 f117zc.sanityn16 -> 1 f117zc.sanityn17 -> 0 f117zc.sanityn18 -> 1 f117zc.sanityn19 -> 1 f117zc.sanityn20 -> 0 f117zc.sanityn21 -> 1 f117zc.sanityn22 -> 0 f117zc.sanityn23 -> 0 f117zc.sanityn24 -> 0 f117zc.sanityn25 -> 1 f117zc.sanityn26 -> 0 f117zc.sanityn27 -> 0 f117zc.sanityn28 -> 1 f117zc.sanityn29 -> 1 f117zc.sanityn30 -> 1 f117zc.sanityn31 -> 0 f117zc.sanityn32 -> 0 f117zc.sanityn33 -> 0 f117zc.sanityn34 -> 0 f117zc.sanityn35 -> 1 f117zc.sanityn36 -> 0 f117zc.sanityn37 -> 1 f117zc.sanityn38 -> 1 f117zc.sanityn39 -> 1 f117zc.sanityn40 -> 1 f117zc.sanityn41 -> 1 f117zc.sanityn42 -> 0 f117zc.sanityn43 -> 1 f117zc.sanityn44 -> 0 f117zc.sanityn45 -> 0 f117zc.sanityn46 -> 0 f117zc.sanityn47 -> 0 f117zc.sanityn48 -> 0 f117zc.sanityn49 -> 1 f117zc.sanityn50 -> 0 f117zc.sanityn51 -> 0 f117zc.sanityn52 -> 0 f117zc.sanityn53 -> 1 f117zc.sanityn54 -> 0 f117zc.sanityn55 -> 1 f117zc.sanityn56 -> 0 f117zc.sanityn57 -> 1 f117zc.sanityn58 -> 1 f117zc.sanityn59 -> 0 f117zc.sanityn60 -> 1 f117zc.sanityn61 -> 0 f117zc.sanityn62 -> 0 f117zc.sanityn63 -> 0 f117zc.sanityn64 -> 1 f117zc.sanityn65 -> 0 f117zc.sanityn66 -> 0 f117zc.sanityn67 -> 1 f117zc.sanityn68 -> 1 f117zc.sanityn69 -> 1 f117zc.sanityn70 -> 0 f117zc.sanityn71 -> 0 f117zc.sanityn72 -> 0 f117zc.sanityn73 -> 0 f117zc.sanityn74 -> 1 f117zc.sanityn75 -> 0 f117zc.sanityn76 -> 1 f117zc.sanityn77 -> 0 f117zc.sanityn78 -> 0 f117zc.sanityn79 -> 1 f117zc.sanityn80 -> 0 f117zc.sanityn81 -> 1 f117zc.sanityn82 -> 1 f117zc.sanityn83 -> 1 f117zc.sanityn84 -> 0 f117zc.sanityn85 -> 1 f117zc.sanityn86 -> 1 f117zc.sanityn87 -> 1 f117zc.sanityn88 -> 0 f117zc.sanityn89 -> 1 f117zc.sanityn90 -> 0 f117zc.sanityn91 -> 1 f117zc.sanityn92 -> 1 f117zc.sanityn93 -> 0 f117zc.sanityn94 -> 0 f117zc.sanityn95 -> 0 f117zc.sanityn96 -> 1 f117zc.sanityn97 -> 0 f117zc.sanityn98 -> 0 f117zc.sanityn99 -> 1 #0 -> 0x200001b70:0xa:0x0 #1 -> 0x2400013a0:0x1:0x0 - unlinked 0 (time 1779075718 ; total 0 ; last 0) total: 100 unlinks in 0 seconds: inf unlinks/second f117zc.sanityn0 f117zc.sanityn1 f117zc.sanityn10 f117zc.sanityn11 f117zc.sanityn12 f117zc.sanityn13 f117zc.sanityn14 f117zc.sanityn15 f117zc.sanityn16 f117zc.sanityn17 f117zc.sanityn18 f117zc.sanityn19 f117zc.sanityn2 f117zc.sanityn20 f117zc.sanityn21 f117zc.sanityn22 f117zc.sanityn23 f117zc.sanityn24 f117zc.sanityn25 f117zc.sanityn26 f117zc.sanityn27 f117zc.sanityn28 f117zc.sanityn29 f117zc.sanityn3 f117zc.sanityn30 f117zc.sanityn31 f117zc.sanityn32 f117zc.sanityn33 f117zc.sanityn34 f117zc.sanityn35 f117zc.sanityn36 f117zc.sanityn37 f117zc.sanityn38 f117zc.sanityn39 f117zc.sanityn4 f117zc.sanityn40 f117zc.sanityn41 f117zc.sanityn42 f117zc.sanityn43 f117zc.sanityn44 f117zc.sanityn45 f117zc.sanityn46 f117zc.sanityn47 f117zc.sanityn48 f117zc.sanityn49 f117zc.sanityn5 f117zc.sanityn50 f117zc.sanityn51 f117zc.sanityn52 f117zc.sanityn53 f117zc.sanityn54 f117zc.sanityn55 f117zc.sanityn56 f117zc.sanityn57 f117zc.sanityn58 f117zc.sanityn59 f117zc.sanityn6 f117zc.sanityn60 f117zc.sanityn61 f117zc.sanityn62 f117zc.sanityn63 f117zc.sanityn64 f117zc.sanityn65 f117zc.sanityn66 f117zc.sanityn67 f117zc.sanityn68 f117zc.sanityn69 f117zc.sanityn7 f117zc.sanityn70 f117zc.sanityn71 f117zc.sanityn72 f117zc.sanityn73 f117zc.sanityn74 f117zc.sanityn75 f117zc.sanityn76 f117zc.sanityn77 f117zc.sanityn78 f117zc.sanityn79 f117zc.sanityn8 f117zc.sanityn80 f117zc.sanityn81 f117zc.sanityn82 f117zc.sanityn83 f117zc.sanityn84 f117zc.sanityn85 f117zc.sanityn86 f117zc.sanityn87 f117zc.sanityn88 f117zc.sanityn89 f117zc.sanityn9 f117zc.sanityn90 f117zc.sanityn91 f117zc.sanityn92 f117zc.sanityn93 f117zc.sanityn94 f117zc.sanityn95 f117zc.sanityn96 f117zc.sanityn97 f117zc.sanityn98 f117zc.sanityn99 f117zc.sanityn10 f117zc.sanityn11 f117zc.sanityn12 f117zc.sanityn13 f117zc.sanityn14 f117zc.sanityn15 f117zc.sanityn16 f117zc.sanityn17 f117zc.sanityn18 f117zc.sanityn19 f117zc.sanityn20 f117zc.sanityn21 f117zc.sanityn22 f117zc.sanityn23 f117zc.sanityn24 f117zc.sanityn25 f117zc.sanityn26 f117zc.sanityn27 f117zc.sanityn28 f117zc.sanityn29 f117zc.sanityn30 f117zc.sanityn31 f117zc.sanityn32 f117zc.sanityn33 f117zc.sanityn34 f117zc.sanityn35 f117zc.sanityn36 f117zc.sanityn37 f117zc.sanityn38 f117zc.sanityn39 f117zc.sanityn40 f117zc.sanityn41 f117zc.sanityn42 f117zc.sanityn43 f117zc.sanityn44 f117zc.sanityn45 f117zc.sanityn46 f117zc.sanityn47 f117zc.sanityn48 f117zc.sanityn49 f117zc.sanityn50 f117zc.sanityn51 f117zc.sanityn52 f117zc.sanityn53 f117zc.sanityn54 f117zc.sanityn55 f117zc.sanityn56 f117zc.sanityn57 f117zc.sanityn58 f117zc.sanityn59 f117zc.sanityn60 f117zc.sanityn61 f117zc.sanityn62 f117zc.sanityn63 f117zc.sanityn64 f117zc.sanityn65 f117zc.sanityn66 f117zc.sanityn67 f117zc.sanityn68 f117zc.sanityn69 f117zc.sanityn70 f117zc.sanityn71 f117zc.sanityn72 f117zc.sanityn73 f117zc.sanityn74 f117zc.sanityn75 f117zc.sanityn76 f117zc.sanityn77 f117zc.sanityn78 f117zc.sanityn79 f117zc.sanityn80 f117zc.sanityn81 f117zc.sanityn82 f117zc.sanityn83 f117zc.sanityn84 f117zc.sanityn85 f117zc.sanityn86 f117zc.sanityn87 f117zc.sanityn88 f117zc.sanityn89 f117zc.sanityn90 f117zc.sanityn91 f117zc.sanityn92 f117zc.sanityn93 f117zc.sanityn94 f117zc.sanityn95 f117zc.sanityn96 f117zc.sanityn97 f117zc.sanityn98 f117zc.sanityn99 mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 117zc (13s) == sanityn test 117zd: TCU: Test striped dir for UID Trash Can type with nodemap ========================================================== 23:42:07 (1779075727) On MGS 192.168.201.135, active = nodemap.active=1 waiting 10 secs for sync On MGS 192.168.201.135, tnm.trusted_nodemap = nodemap.tnm.trusted_nodemap=1 waiting 10 secs for sync nodemap.tnm.idmap= [ { idtype: uid, client_id: 500, fs_id: 10000 }, { idtype: uid, client_id: 501, fs_id: 10001 }, { idtype: uid, client_id: 502, fs_id: 10002 }, { idtype: uid, client_id: 503, fs_id: 10003 }, { idtype: uid, client_id: 504, fs_id: 10004 }, { idtype: uid, client_id: 505, fs_id: 10005 }, { idtype: uid, client_id: 506, fs_id: 10006 }, { idtype: uid, client_id: 507, fs_id: 10007 }, { idtype: uid, client_id: 508, fs_id: 10008 }, { idtype: uid, client_id: 509, fs_id: 10009 } ] mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid Trash Can Type: uid Creating 100 files under striped directory: /mnt/lustre/d117zd.sanityn/striped_dir total: 100 open/close in 0.32 seconds: 313.01 ops/second f117zd.sanityn0 -> 0 f117zd.sanityn1 -> 0 f117zd.sanityn2 -> 0 f117zd.sanityn3 -> 0 f117zd.sanityn4 -> 0 f117zd.sanityn5 -> 0 f117zd.sanityn6 -> 0 f117zd.sanityn7 -> 0 f117zd.sanityn8 -> 0 f117zd.sanityn9 -> 0 f117zd.sanityn10 -> 0 f117zd.sanityn11 -> 1 f117zd.sanityn12 -> 1 f117zd.sanityn13 -> 0 f117zd.sanityn14 -> 1 f117zd.sanityn15 -> 1 f117zd.sanityn16 -> 1 f117zd.sanityn17 -> 1 f117zd.sanityn18 -> 1 f117zd.sanityn19 -> 0 f117zd.sanityn20 -> 1 f117zd.sanityn21 -> 0 f117zd.sanityn22 -> 0 f117zd.sanityn23 -> 1 f117zd.sanityn24 -> 1 f117zd.sanityn25 -> 0 f117zd.sanityn26 -> 1 f117zd.sanityn27 -> 1 f117zd.sanityn28 -> 1 f117zd.sanityn29 -> 0 f117zd.sanityn30 -> 1 f117zd.sanityn31 -> 0 f117zd.sanityn32 -> 1 f117zd.sanityn33 -> 0 f117zd.sanityn34 -> 1 f117zd.sanityn35 -> 0 f117zd.sanityn36 -> 0 f117zd.sanityn37 -> 0 f117zd.sanityn38 -> 0 f117zd.sanityn39 -> 1 f117zd.sanityn40 -> 1 f117zd.sanityn41 -> 0 f117zd.sanityn42 -> 1 f117zd.sanityn43 -> 1 f117zd.sanityn44 -> 0 f117zd.sanityn45 -> 0 f117zd.sanityn46 -> 1 f117zd.sanityn47 -> 0 f117zd.sanityn48 -> 1 f117zd.sanityn49 -> 0 f117zd.sanityn50 -> 1 f117zd.sanityn51 -> 1 f117zd.sanityn52 -> 1 f117zd.sanityn53 -> 0 f117zd.sanityn54 -> 0 f117zd.sanityn55 -> 0 f117zd.sanityn56 -> 0 f117zd.sanityn57 -> 1 f117zd.sanityn58 -> 1 f117zd.sanityn59 -> 1 f117zd.sanityn60 -> 1 f117zd.sanityn61 -> 0 f117zd.sanityn62 -> 0 f117zd.sanityn63 -> 0 f117zd.sanityn64 -> 0 f117zd.sanityn65 -> 1 f117zd.sanityn66 -> 0 f117zd.sanityn67 -> 0 f117zd.sanityn68 -> 1 f117zd.sanityn69 -> 1 f117zd.sanityn70 -> 1 f117zd.sanityn71 -> 1 f117zd.sanityn72 -> 0 f117zd.sanityn73 -> 0 f117zd.sanityn74 -> 0 f117zd.sanityn75 -> 0 f117zd.sanityn76 -> 1 f117zd.sanityn77 -> 1 f117zd.sanityn78 -> 1 f117zd.sanityn79 -> 0 f117zd.sanityn80 -> 0 f117zd.sanityn81 -> 0 f117zd.sanityn82 -> 1 f117zd.sanityn83 -> 1 f117zd.sanityn84 -> 1 f117zd.sanityn85 -> 0 f117zd.sanityn86 -> 0 f117zd.sanityn87 -> 1 f117zd.sanityn88 -> 0 f117zd.sanityn89 -> 0 f117zd.sanityn90 -> 0 f117zd.sanityn91 -> 0 f117zd.sanityn92 -> 1 f117zd.sanityn93 -> 1 f117zd.sanityn94 -> 0 f117zd.sanityn95 -> 1 f117zd.sanityn96 -> 0 f117zd.sanityn97 -> 1 f117zd.sanityn98 -> 1 f117zd.sanityn99 -> 1 #0 -> 0x200001b70:0xe:0x0 #1 -> 0x2400013a0:0x4:0x0 - unlinked 0 (time 1779075755 ; total 0 ; last 0) total: 100 unlinks in 0 seconds: inf unlinks/second f117zd.sanityn0 f117zd.sanityn1 f117zd.sanityn10 f117zd.sanityn11 f117zd.sanityn12 f117zd.sanityn13 f117zd.sanityn14 f117zd.sanityn15 f117zd.sanityn16 f117zd.sanityn17 f117zd.sanityn18 f117zd.sanityn19 f117zd.sanityn2 f117zd.sanityn20 f117zd.sanityn21 f117zd.sanityn22 f117zd.sanityn23 f117zd.sanityn24 f117zd.sanityn25 f117zd.sanityn26 f117zd.sanityn27 f117zd.sanityn28 f117zd.sanityn29 f117zd.sanityn3 f117zd.sanityn30 f117zd.sanityn31 f117zd.sanityn32 f117zd.sanityn33 f117zd.sanityn34 f117zd.sanityn35 f117zd.sanityn36 f117zd.sanityn37 f117zd.sanityn38 f117zd.sanityn39 f117zd.sanityn4 f117zd.sanityn40 f117zd.sanityn41 f117zd.sanityn42 f117zd.sanityn43 f117zd.sanityn44 f117zd.sanityn45 f117zd.sanityn46 f117zd.sanityn47 f117zd.sanityn48 f117zd.sanityn49 f117zd.sanityn5 f117zd.sanityn50 f117zd.sanityn51 f117zd.sanityn52 f117zd.sanityn53 f117zd.sanityn54 f117zd.sanityn55 f117zd.sanityn56 f117zd.sanityn57 f117zd.sanityn58 f117zd.sanityn59 f117zd.sanityn6 f117zd.sanityn60 f117zd.sanityn61 f117zd.sanityn62 f117zd.sanityn63 f117zd.sanityn64 f117zd.sanityn65 f117zd.sanityn66 f117zd.sanityn67 f117zd.sanityn68 f117zd.sanityn69 f117zd.sanityn7 f117zd.sanityn70 f117zd.sanityn71 f117zd.sanityn72 f117zd.sanityn73 f117zd.sanityn74 f117zd.sanityn75 f117zd.sanityn76 f117zd.sanityn77 f117zd.sanityn78 f117zd.sanityn79 f117zd.sanityn8 f117zd.sanityn80 f117zd.sanityn81 f117zd.sanityn82 f117zd.sanityn83 f117zd.sanityn84 f117zd.sanityn85 f117zd.sanityn86 f117zd.sanityn87 f117zd.sanityn88 f117zd.sanityn89 f117zd.sanityn9 f117zd.sanityn90 f117zd.sanityn91 f117zd.sanityn92 f117zd.sanityn93 f117zd.sanityn94 f117zd.sanityn95 f117zd.sanityn96 f117zd.sanityn97 f117zd.sanityn98 f117zd.sanityn99 f117zd.sanityn10 f117zd.sanityn11 f117zd.sanityn12 f117zd.sanityn13 f117zd.sanityn14 f117zd.sanityn15 f117zd.sanityn16 f117zd.sanityn17 f117zd.sanityn18 f117zd.sanityn19 f117zd.sanityn20 f117zd.sanityn21 f117zd.sanityn22 f117zd.sanityn23 f117zd.sanityn24 f117zd.sanityn25 f117zd.sanityn26 f117zd.sanityn27 f117zd.sanityn28 f117zd.sanityn29 f117zd.sanityn30 f117zd.sanityn31 f117zd.sanityn32 f117zd.sanityn33 f117zd.sanityn34 f117zd.sanityn35 f117zd.sanityn36 f117zd.sanityn37 f117zd.sanityn38 f117zd.sanityn39 f117zd.sanityn40 f117zd.sanityn41 f117zd.sanityn42 f117zd.sanityn43 f117zd.sanityn44 f117zd.sanityn45 f117zd.sanityn46 f117zd.sanityn47 f117zd.sanityn48 f117zd.sanityn49 f117zd.sanityn50 f117zd.sanityn51 f117zd.sanityn52 f117zd.sanityn53 f117zd.sanityn54 f117zd.sanityn55 f117zd.sanityn56 f117zd.sanityn57 f117zd.sanityn58 f117zd.sanityn59 f117zd.sanityn60 f117zd.sanityn61 f117zd.sanityn62 f117zd.sanityn63 f117zd.sanityn64 f117zd.sanityn65 f117zd.sanityn66 f117zd.sanityn67 f117zd.sanityn68 f117zd.sanityn69 f117zd.sanityn70 f117zd.sanityn71 f117zd.sanityn72 f117zd.sanityn73 f117zd.sanityn74 f117zd.sanityn75 f117zd.sanityn76 f117zd.sanityn77 f117zd.sanityn78 f117zd.sanityn79 f117zd.sanityn80 f117zd.sanityn81 f117zd.sanityn82 f117zd.sanityn83 f117zd.sanityn84 f117zd.sanityn85 f117zd.sanityn86 f117zd.sanityn87 f117zd.sanityn88 f117zd.sanityn89 f117zd.sanityn90 f117zd.sanityn91 f117zd.sanityn92 f117zd.sanityn93 f117zd.sanityn94 f117zd.sanityn95 f117zd.sanityn96 f117zd.sanityn97 f117zd.sanityn98 f117zd.sanityn99 mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid On MGS 192.168.201.135, active = nodemap.active=0 waiting 10 secs for sync PASS 117zd (49s) == sanityn test 118Aa: TCU: Orphan object handling for a regular file or an empty dir ========================================================== 23:42:56 (1779075776) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain oleg135-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg135-server: debugfs 1.47.3-wc2 (11-Nov-2025) 64004 (12) . 2 (4084) .. 520188 (4096) 0x200001b71:0x78:0x0 Waiting for MDT destroys to complete oleg135-server: debugfs 1.47.3-wc2 (11-Nov-2025) 64004 (12) . 2 (4084) .. 0 (4096) 0x200001b71:0x78:0x0 sanityn test_118Aa: @@@@@@ FAIL: 0x200001b71:0x78:0x0 should be removed from PENDING Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7318:error() = /home/green/git/lustre-release/lustre/tests/sanityn.sh:9231:test_118Aa() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7694:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7757:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7572:run_test() = /home/green/git/lustre-release/lustre/tests/sanityn.sh:9253:main() Dumping lctl log to /tmp/testlogs//sanityn.test_118Aa.*.1779075794.log rsync: chown "/tmp/testlogs/.sanityn.test_118Aa.debug_log.oleg135-server.1779075794.log.cYBVP1" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanityn.test_118Aa.dmesg.oleg135-server.1779075794.log.AoxGUk" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid FAIL 118Aa (21s) == sanityn test 118Ab: TCU: Orphan object handling for non-emtpy dir ========================================================== 23:43:17 (1779075797) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain oleg135-server: debugfs 1.47.3-wc2 (11-Nov-2025) 520190 (4096) 0x200001b71:0x7a:0x0 Waiting for MDT destroys to complete oleg135-server: debugfs 1.47.3-wc2 (11-Nov-2025) 0 (4096) 0x200001b71:0x7a:0x0 sanityn test_118Ab: @@@@@@ FAIL: 0x200001b71:0x7a:0x0 should be removed/cleanup from PENDING Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7318:error() = /home/green/git/lustre-release/lustre/tests/sanityn.sh:9297:test_118Ab() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7694:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7757:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7572:run_test() = /home/green/git/lustre-release/lustre/tests/sanityn.sh:9301:main() Dumping lctl log to /tmp/testlogs//sanityn.test_118Ab.*.1779075810.log rsync: chown "/tmp/testlogs/.sanityn.test_118Ab.debug_log.oleg135-server.1779075810.log.0IjM4p" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanityn.test_118Ab.dmesg.oleg135-server.1779075810.log.hintOO" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid FAIL 118Ab (15s) == sanityn test 118Ba: TCU: 'oldest' repeated deletion for a regular file or empty dir ========================================================== 23:43:33 (1779075813) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain mdd.lustre-MDT0000.trash_can_delete_policy=oldest mdd.lustre-MDT0001.trash_can_delete_policy=oldest llite.lustre-ffffa0e704f1e000.trash_file_read=1 llite.lustre-ffffa0e7050cf000.trash_file_read=1 llite.lustre-ffffa0e704f1e000.trash_file_read=0 llite.lustre-ffffa0e7050cf000.trash_file_read=0 mdd.lustre-MDT0000.trash_can_delete_policy=timestamp mdd.lustre-MDT0001.trash_can_delete_policy=timestamp mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 118Ba (9s) == sanityn test 118Bb: TCU: 'timestamp' repeated deletion policy ========================================================== 23:43:41 (1779075821) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain mdd.lustre-MDT0000.trash_can_delete_policy=timestamp mdd.lustre-MDT0001.trash_can_delete_policy=timestamp f118Bb.sanityn f118Bb.sanityn.2026-05-18T03:43:45 f118Bb.sanityn.2026-05-18T03:43:45.796522508 f118Bb.sanityn.2026-05-18T03:43:45.822629956 f118Bb.sanityn.2026-05-18T03:43:45.848742857 f118Bb.sanityn.2026-05-18T03:43:45.875542168 f118Bb.sanityn.2026-05-18T03:43:45.901300059 f118Bb.sanityn.2026-05-18T03:43:45.927211119 f118Bb.sanityn.2026-05-18T03:43:45.953469270 f118Bb.sanityn.2026-05-18T03:43:45.979376134 f118Bb.sanityn.2026-05-18T03:43:46 f118Bb.sanityn.2026-05-18T03:43:46.031273676 f118Bb.sanityn.2026-05-18T03:43:46.056786483 f118Bb.sanityn.2026-05-18T03:43:46.083201931 f118Bb.sanityn.2026-05-18T03:43:46.109323639 f118Bb.sanityn.2026-05-18T03:43:46.135219306 d118Bb.sanityn d118Bb.sanityn.2026-05-18T03:43:46 d118Bb.sanityn.2026-05-18T03:43:46.789646822 d118Bb.sanityn.2026-05-18T03:43:46.808534856 d118Bb.sanityn.2026-05-18T03:43:46.828275161 d118Bb.sanityn.2026-05-18T03:43:46.847336073 d118Bb.sanityn.2026-05-18T03:43:46.866771304 d118Bb.sanityn.2026-05-18T03:43:46.886100299 d118Bb.sanityn.2026-05-18T03:43:46.905274590 d118Bb.sanityn.2026-05-18T03:43:46.924676405 d118Bb.sanityn.2026-05-18T03:43:46.943645886 d118Bb.sanityn.2026-05-18T03:43:46.963037071 d118Bb.sanityn.2026-05-18T03:43:46.982155642 d118Bb.sanityn.2026-05-18T03:43:47 d118Bb.sanityn.2026-05-18T03:43:47.020232715 d118Bb.sanityn.2026-05-18T03:43:47.039437606 d118Bb.sanityn.mulevel d118Bb.sanityn.mulevel.2026-05-18T03:43:48 d118Bb.sanityn.mulevel.2026-05-18T03:43:48.168172279 d118Bb.sanityn.mulevel.2026-05-18T03:43:48.296830693 d118Bb.sanityn.mulevel.2026-05-18T03:43:48.427567751 d118Bb.sanityn.mulevel.2026-05-18T03:43:48.555456121 d118Bb.sanityn.mulevel.2026-05-18T03:43:48.683466684 d118Bb.sanityn.mulevel.2026-05-18T03:43:48.810413940 d118Bb.sanityn.mulevel.2026-05-18T03:43:48.937424766 d118Bb.sanityn.mulevel.2026-05-18T03:43:49 d118Bb.sanityn.mulevel.2026-05-18T03:43:49.186775110 d118Bb.sanityn.mulevel.2026-05-18T03:43:49.312782663 d118Bb.sanityn.mulevel.2026-05-18T03:43:49.440915120 d118Bb.sanityn.mulevel.2026-05-18T03:43:49.565926868 d118Bb.sanityn.mulevel.2026-05-18T03:43:49.690485867 d118Bb.sanityn.mulevel.2026-05-18T03:43:49.813763981 mdd.lustre-MDT0000.trash_can_delete_policy=timestamp mdd.lustre-MDT0001.trash_can_delete_policy=timestamp mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 118Bb (13s) == sanityn test 118C: TCU: add support for rename() ====== 23:43:54 (1779075834) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain mdd.lustre-MDT0000.trash_can_delete_policy=timestamp mdd.lustre-MDT0001.trash_can_delete_policy=timestamp f118C.sanityn f118C.sanityn f118C.sanityn f118C.sanityn.2026-05-18T03:43:56 f118C.sanityn.2026-05-18T03:43:56.435904893 mdd.lustre-MDT0000.trash_can_delete_policy=timestamp mdd.lustre-MDT0001.trash_can_delete_policy=timestamp mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 118C (5s) == sanityn test 118d: TCU: repeated deletion of same fname for a empty dir in DNE env ========================================================== 23:43:59 (1779075839) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain mdd.lustre-MDT0000.trash_can_delete_policy=timestamp mdd.lustre-MDT0001.trash_can_delete_policy=timestamp d118d.sanityn.0 d118d.sanityn.0.2026-05-18T03:44:01 d118d.sanityn.0.2026-05-18T03:44:01.731720097 d118d.sanityn.0.2026-05-18T03:44:01.750887015 d118d.sanityn.0.2026-05-18T03:44:01.774017699 d118d.sanityn.0.2026-05-18T03:44:01.792788144 d118d.sanityn.0.2026-05-18T03:44:01.817176969 d118d.sanityn.0.2026-05-18T03:44:01.835364308 d118d.sanityn.0.2026-05-18T03:44:01.859184687 d118d.sanityn.0.2026-05-18T03:44:01.877739009 mdd.lustre-MDT0000.trash_can_delete_policy=timestamp mdd.lustre-MDT0001.trash_can_delete_policy=timestamp mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 118d (6s) == sanityn test 118e: TCU: Repeated deletion of same name for a non-empty remote dir ========================================================== 23:44:05 (1779075845) SKIP: sanityn test_118e needs 3 MDTs at least SKIP 118e (1s) == sanityn test 118f: Avoid caching negative dentry cache for dir in Trash Can ========================================================== 23:44:06 (1779075846) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 9539: tree: command not found stat: cannot statx '/mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0xf0:0x0/f118f.sanityn.1': No such file or directory === 2nd rm /mnt/lustre/d118f.sanityn/f118f.sanityn.1 ==== total 8 drwxr-xr-x 2 root root 4096 May 17 23:44 . d--------- 32 root root 4096 May 17 21:23 .. -rw-r--r-- 1 root root 0 May 17 23:44 f118f.sanityn.1 mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 118f (4s) == sanityn test 118g: TCU: Tolerate ENOSPC error bypass Trash Can ========================================================== 23:44:10 (1779075850) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1024000 13469 1010531 2% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1024000 9340 1014660 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 262144 3950 258194 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 262144 3950 258194 2% /mnt/lustre[OST:1] filesystem_summary: 539197 22809 516388 5% /mnt/lustre - create 7902 (time 1779075863.16 total 10.00 last 790.18) - create 10000 (time 1779075865.75 total 12.59 last 811.28) - create 17665 (time 1779075875.75 total 22.59 last 766.43) - create 20000 (time 1779075878.73 total 25.57 last 782.94) - create 29088 (time 1779075888.73 total 35.57 last 908.62) - create 30000 (time 1779075890.75 total 37.59 last 452.67) - create 34990 (time 1779075900.75 total 47.59 last 499.00) - create 39707 (time 1779075910.75 total 57.59 last 471.59) - create 44501 (time 1779075920.75 total 67.59 last 479.38) - create 48830 (time 1779075930.75 total 77.59 last 432.84) - create 50000 (time 1779075933.48 total 80.32 last 428.75) - create 54491 (time 1779075943.48 total 90.32 last 449.05) - create 59064 (time 1779075953.48 total 100.32 last 457.29) - create 60000 (time 1779075955.53 total 102.37 last 456.94) - create 64771 (time 1779075965.53 total 112.37 last 477.06) total: 68306 create in 120.00 seconds: 569.22 ops/second - unlinked 0 (time 1779075975 ; total 0 ; last 0) - unlinked 10000 (time 1779076011 ; total 36 ; last 36) - unlinked 20000 (time 1779076042 ; total 67 ; last 31) - unlinked 30000 (time 1779076082 ; total 107 ; last 40) - unlinked 40000 (time 1779076112 ; total 137 ; last 30) - unlinked 50000 (time 1779076136 ; total 161 ; last 24) - unlinked 60000 (time 1779076168 ; total 193 ; last 32) unlink(/mnt/lustre/d118g.sanityn/tf68306) error: No such file or directory total: 68306 unlinks in 226 seconds: 302.238953 unlinks/second - create 9545 (time 1779076498.60 total 10.00 last 954.46) - create 18856 (time 1779076508.60 total 20.00 last 931.08) - create 28586 (time 1779076518.60 total 30.00 last 972.99) - create 38104 (time 1779076528.60 total 40.00 last 951.72) - create 47921 (time 1779076538.60 total 50.00 last 981.70) - create 50000 (time 1779076540.77 total 52.17 last 959.40) - create 59678 (time 1779076550.77 total 62.17 last 967.73) - create 69360 (time 1779076560.77 total 72.17 last 968.17) - create 79105 (time 1779076570.77 total 82.17 last 974.45) - create 86793 (time 1779076580.77 total 92.17 last 768.80) - create 90000 (time 1779076585.34 total 96.74 last 702.16) - create 98096 (time 1779076595.34 total 106.74 last 809.59) - create 100000 (time 1779076597.94 total 109.34 last 731.70) - create 108618 (time 1779076607.94 total 119.34 last 861.72) total: 109111 create in 120.00 seconds: 909.25 ops/second - unlinked 0 (time 1779076611 ; total 0 ; last 0) - unlinked 10000 (time 1779076641 ; total 30 ; last 30) - unlinked 20000 (time 1779076667 ; total 56 ; last 26) - unlinked 30000 (time 1779076695 ; total 84 ; last 28) - unlinked 40000 (time 1779076718 ; total 107 ; last 23) - unlinked 50000 (time 1779076747 ; total 136 ; last 29) - unlinked 60000 (time 1779076787 ; total 176 ; last 40) - unlinked 70000 (time 1779076818 ; total 207 ; last 31) - unlinked 80000 (time 1779076855 ; total 244 ; last 37) - unlinked 90000 (time 1779076898 ; total 287 ; last 43) - unlinked 100000 (time 1779076940 ; total 329 ; last 42) unlink(/mnt/lustre/d118g.sanityn/tf109111) error: No such file or directory total: 109111 unlinks in 369 seconds: 295.693756 unlinks/second mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 118g (1755s) == sanityn test 118h: TCU: run mdtest with TCU enabled === 00:13:25 (1779077605) SKIP: sanityn test_118h mdtest not found SKIP 118h (2s) == sanityn test 118i: TCU: trusted.unrm xattr should not be cached on client-side ========================================================== 00:13:27 (1779077607) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain total 8 drwxr-xr-x 2 root root 4096 May 18 00:13 . d--------- 43 root root 4096 May 17 21:23 .. -rw-r--r-- 1 root root 0 May 18 00:13 f118i.sanityn getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/d118i.sanityn user.test="test_value" user.test access RPC count: 0 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/.lustre/.ltrash/MDT0000/0x200000007:0x1:0x0/d118i.sanityn trusted.unrm=0sAAAAAAAAAAAAAAAAAAAAAOuRCmoAAAAA trusted.unrm first access RPC count: 2 mdd.lustre-MDT0000.trash_can_delete_policy=timestamp mdd.lustre-MDT0001.trash_can_delete_policy=timestamp mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 118i (9s) == sanityn test 118j: TCU: access .Trash from users with different UID ========================================================== 00:13:36 (1779077616) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid Trash Can Type: uid mdd.lustre-MDT0000.trash_can_puid=1 mdd.lustre-MDT0001.trash_can_puid=1 File: /mnt/lustre/d118j.sanityn Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115305969399299 Links: 2 Access: (0777/drwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-18 00:13:41.000000000 -0400 Modify: 2026-05-18 00:13:41.000000000 -0400 Change: 2026-05-18 00:13:41.000000000 -0400 Birth: 2026-05-18 00:13:41.000000000 -0400 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d118j.sanityn/f118j.sanityn] -rw-r--r-- 1 sanityusr sanityusr 0 May 18 00:13 /mnt/lustre/d118j.sanityn/f118j.sanityn running as uid/gid/euid/egid 500/500/500/500, groups: 500 [unlink] [/mnt/lustre/d118j.sanityn/f118j.sanityn] f118j.sanityn Trash files under /mnt/lustre/.lustre/trash/MDT0000 /mnt/lustre/.lustre/.ltrash/MDT0000: 0 0x200000007:0x1:0x0 0x200000bd0:0x657:0x0 0x200000bd1:0xbcc:0x0 0x200000bd1:0xbd2:0x0 0x200000bd1:0xbe3:0x0 0x200000bd1:0xbe7:0x0 0x200000bd1:0xbec:0x0 0x200000bd1:0xbf3:0x0 0x200000bd1:0xc06:0x0 0x200000bd1:0xc0c:0x0 0x200000bd1:0xc12:0x0 0x200000bd1:0xc1c:0x0 0x200000bd1:0xc1f:0x0 0x200000bd1:0xc25:0x0 0x200000bd1:0xc28:0x0 0x200000bd1:0xc2e:0x0 0x200000bd1:0xc33:0x0 0x200000bd1:0xc37:0x0 0x200000bd1:0xc39:0x0 0x200000bd1:0xc93:0x0 0x200000bd1:0xc95:0x0 0x2000013a1:0x3:0x0 0x2000013a1:0x3ee:0x0 0x2000013a1:0x3f3:0x0 0x2000013a1:0x3f8:0x0 0x2000013a1:0x401:0x0 0x2000013a1:0x5:0x0 0x200001b71:0x10bc7:0x0 0x200001b71:0x1:0x0 0x200001b71:0x77:0x0 0x200001b71:0x79:0x0 0x200001b71:0x7e:0x0 0x200001b71:0x84:0x0 0x200001b71:0xa:0x0 0x200001b71:0xe5:0x0 0x200001b71:0xea:0x0 0x200001b71:0xf0:0x0 500 tnm /mnt/lustre/.lustre/.ltrash/MDT0000/0: 0x200001b73:0xb603:0x0 /mnt/lustre/.lustre/.ltrash/MDT0000/0/0x200001b73:0xb603:0x0: f118j.sanityn /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000007:0x1:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd0:0x657:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbcc:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbd2:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbec:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc06:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc0c:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc12:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc1c:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc1f:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc25:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc28:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc2e:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc33:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc37:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc39:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc93:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc95:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3ee:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3f3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3f8:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x401:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x5:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0x10bc7:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0x1:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0x77:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0x79:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0x7e:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0x84:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0xa:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0xe5:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0xea:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0xf0:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/500: /mnt/lustre/.lustre/.ltrash/MDT0000/tnm: /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 9690: tree: command not found running as uid/gid/euid/egid 500/500/500/500, groups: 500 [mkdir] [/mnt/lustre/d118j.sanityn/d118j.sanityn.1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d118j.sanityn/d118j.sanityn.1/f118j.sanityn.1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [unlink] [/mnt/lustre/d118j.sanityn/d118j.sanityn.1/f118j.sanityn.1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [rmdir] [/mnt/lustre/d118j.sanityn/d118j.sanityn.1] /mnt/lustre/d118j.sanityn/.Trash: d118j.sanityn.1 f118j.sanityn /mnt/lustre/d118j.sanityn/.Trash/d118j.sanityn.1: f118j.sanityn.1 /mnt/lustre/.lustre/.ltrash/MDT0000: 0 0x200000007:0x1:0x0 0x200000bd0:0x657:0x0 0x200000bd1:0xbcc:0x0 0x200000bd1:0xbd2:0x0 0x200000bd1:0xbe3:0x0 0x200000bd1:0xbe7:0x0 0x200000bd1:0xbec:0x0 0x200000bd1:0xbf3:0x0 0x200000bd1:0xc06:0x0 0x200000bd1:0xc0c:0x0 0x200000bd1:0xc12:0x0 0x200000bd1:0xc1c:0x0 0x200000bd1:0xc1f:0x0 0x200000bd1:0xc25:0x0 0x200000bd1:0xc28:0x0 0x200000bd1:0xc2e:0x0 0x200000bd1:0xc33:0x0 0x200000bd1:0xc37:0x0 0x200000bd1:0xc39:0x0 0x200000bd1:0xc93:0x0 0x200000bd1:0xc95:0x0 0x2000013a1:0x3:0x0 0x2000013a1:0x3ee:0x0 0x2000013a1:0x3f3:0x0 0x2000013a1:0x3f8:0x0 0x2000013a1:0x401:0x0 0x2000013a1:0x5:0x0 0x200001b71:0x10bc7:0x0 0x200001b71:0x1:0x0 0x200001b71:0x77:0x0 0x200001b71:0x79:0x0 0x200001b71:0x7e:0x0 0x200001b71:0x84:0x0 0x200001b71:0xa:0x0 0x200001b71:0xe5:0x0 0x200001b71:0xea:0x0 0x200001b71:0xf0:0x0 500 tnm /mnt/lustre/.lustre/.ltrash/MDT0000/0: 0x200001b73:0xb603:0x0 /mnt/lustre/.lustre/.ltrash/MDT0000/0/0x200001b73:0xb603:0x0: d118j.sanityn.1 f118j.sanityn /mnt/lustre/.lustre/.ltrash/MDT0000/0/0x200001b73:0xb603:0x0/d118j.sanityn.1: f118j.sanityn.1 /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000007:0x1:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd0:0x657:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbcc:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbd2:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbec:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc06:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc0c:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc12:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc1c:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc1f:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc25:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc28:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc2e:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc33:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc37:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc39:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc93:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc95:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3ee:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3f3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3f8:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x401:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x5:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0x10bc7:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0x1:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0x77:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0x79:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0x7e:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0x84:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0xa:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0xe5:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0xea:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0xf0:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/500: /mnt/lustre/.lustre/.ltrash/MDT0000/tnm: /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 9699: tree: command not found f118j.sanityn d118j.sanityn.1 f118j.sanityn.1 mdd.lustre-MDT0000.trash_can_puid=0 mdd.lustre-MDT0001.trash_can_puid=0 mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 118j (11s) == sanityn test 118k: TCU: acess .Trash from users with different NODEMAP UIDs ========================================================== 00:13:47 (1779077627) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid Trash Can Type: uid mdd.lustre-MDT0000.trash_can_puid=1 mdd.lustre-MDT0001.trash_can_puid=1 On MGS 192.168.201.135, active = nodemap.active=1 waiting 10 secs for sync On MGS 192.168.201.135, tnm.trusted_nodemap = nodemap.tnm.trusted_nodemap=1 waiting 10 secs for sync nodemap.tnm.idmap= [ { idtype: uid, client_id: 500, fs_id: 10000 }, { idtype: uid, client_id: 501, fs_id: 10001 }, { idtype: uid, client_id: 502, fs_id: 10002 }, { idtype: uid, client_id: 503, fs_id: 10003 }, { idtype: uid, client_id: 504, fs_id: 10004 }, { idtype: uid, client_id: 505, fs_id: 10005 }, { idtype: uid, client_id: 506, fs_id: 10006 }, { idtype: uid, client_id: 507, fs_id: 10007 }, { idtype: uid, client_id: 508, fs_id: 10008 }, { idtype: uid, client_id: 509, fs_id: 10009 } ] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d118k.sanityn/f118k.sanityn] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [unlink] [/mnt/lustre/d118k.sanityn/f118k.sanityn] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [mkdir] [/mnt/lustre/d118k.sanityn/d118k.sanityn.1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d118k.sanityn/d118k.sanityn.1/f118k.sanityn.1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [unlink] [/mnt/lustre/d118k.sanityn/d118k.sanityn.1/f118k.sanityn.1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [rmdir] [/mnt/lustre/d118k.sanityn/d118k.sanityn.1] /mnt/lustre/.lustre/.ltrash: MDT0000 MDT0001 /mnt/lustre/.lustre/.ltrash/MDT0000: 0 0x200000007:0x1:0x0 0x200000bd0:0x657:0x0 0x200000bd1:0xbcc:0x0 0x200000bd1:0xbd2:0x0 0x200000bd1:0xbe3:0x0 0x200000bd1:0xbe7:0x0 0x200000bd1:0xbec:0x0 0x200000bd1:0xbf3:0x0 0x200000bd1:0xc06:0x0 0x200000bd1:0xc0c:0x0 0x200000bd1:0xc12:0x0 0x200000bd1:0xc1c:0x0 0x200000bd1:0xc1f:0x0 0x200000bd1:0xc25:0x0 0x200000bd1:0xc28:0x0 0x200000bd1:0xc2e:0x0 0x200000bd1:0xc33:0x0 0x200000bd1:0xc37:0x0 0x200000bd1:0xc39:0x0 0x200000bd1:0xc93:0x0 0x200000bd1:0xc95:0x0 0x2000013a1:0x3:0x0 0x2000013a1:0x3ee:0x0 0x2000013a1:0x3f3:0x0 0x2000013a1:0x3f8:0x0 0x2000013a1:0x401:0x0 0x2000013a1:0x5:0x0 0x200001b71:0x10bc7:0x0 0x200001b71:0x1:0x0 0x200001b71:0x77:0x0 0x200001b71:0x79:0x0 0x200001b71:0x7e:0x0 0x200001b71:0x84:0x0 0x200001b71:0xa:0x0 0x200001b71:0xe5:0x0 0x200001b71:0xea:0x0 0x200001b71:0xf0:0x0 500 tnm /mnt/lustre/.lustre/.ltrash/MDT0000/0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000007:0x1:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd0:0x657:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbcc:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbd2:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbec:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc06:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc0c:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc12:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc1c:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc1f:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc25:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc28:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc2e:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc33:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc37:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc39:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc93:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc95:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3ee:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3f3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3f8:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x401:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x5:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0x10bc7:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0x1:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0x77:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0x79:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0x7e:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0x84:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0xa:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0xe5:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0xea:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0xf0:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/500: /mnt/lustre/.lustre/.ltrash/MDT0000/tnm: 0 500 /mnt/lustre/.lustre/.ltrash/MDT0000/tnm/0: 0x200001b73:0xb607:0x0 /mnt/lustre/.lustre/.ltrash/MDT0000/tnm/0/0x200001b73:0xb607:0x0: d118k.sanityn.1 f118k.sanityn /mnt/lustre/.lustre/.ltrash/MDT0000/tnm/0/0x200001b73:0xb607:0x0/d118k.sanityn.1: f118k.sanityn.1 /mnt/lustre/.lustre/.ltrash/MDT0000/tnm/500: /mnt/lustre/.lustre/.ltrash/MDT0001: 0 0x240000404:0x5af:0x0 0x240000404:0x5b1:0x0 0x240000405:0x41:0x0 0x240000405:0xc:0x0 tnm /mnt/lustre/.lustre/.ltrash/MDT0001/0: /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000404:0x5af:0x0: /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000404:0x5b1:0x0: /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000405:0x41:0x0: /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000405:0xc:0x0: /mnt/lustre/.lustre/.ltrash/MDT0001/tnm: Access from .Trash: /mnt/lustre/d118k.sanityn/.Trash: d118k.sanityn.1 f118k.sanityn /mnt/lustre/d118k.sanityn/.Trash/d118k.sanityn.1: f118k.sanityn.1 f118k.sanityn d118k.sanityn.1 f118k.sanityn.1 On MGS 192.168.201.135, active = nodemap.active=0 waiting 10 secs for sync mdd.lustre-MDT0000.trash_can_puid=0 mdd.lustre-MDT0001.trash_can_puid=0 mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 118k (49s) == sanityn test 118l: Access global trash dir via LU_TRASH_FID with subdir mount ========================================================== 00:14:37 (1779077677) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre/d118l.sanityn /mnt/lustre_d118l.sanityn /mnt/lustre/.lustre/.ltrash: MDT0000 MDT0001 /mnt/lustre/.lustre/.ltrash/MDT0000: 0 0x200000007:0x1:0x0 0x200000bd0:0x657:0x0 0x200000bd1:0xbcc:0x0 0x200000bd1:0xbd2:0x0 0x200000bd1:0xbe3:0x0 0x200000bd1:0xbe7:0x0 0x200000bd1:0xbec:0x0 0x200000bd1:0xbf3:0x0 0x200000bd1:0xc06:0x0 0x200000bd1:0xc0c:0x0 0x200000bd1:0xc12:0x0 0x200000bd1:0xc1c:0x0 0x200000bd1:0xc1f:0x0 0x200000bd1:0xc25:0x0 0x200000bd1:0xc28:0x0 0x200000bd1:0xc2e:0x0 0x200000bd1:0xc33:0x0 0x200000bd1:0xc37:0x0 0x200000bd1:0xc39:0x0 0x200000bd1:0xc93:0x0 0x200000bd1:0xc95:0x0 0x2000013a1:0x3:0x0 0x2000013a1:0x3ee:0x0 0x2000013a1:0x3f3:0x0 0x2000013a1:0x3f8:0x0 0x2000013a1:0x401:0x0 0x2000013a1:0x5:0x0 0x200001b71:0x10bc7:0x0 0x200001b71:0x1:0x0 0x200001b71:0x77:0x0 0x200001b71:0x79:0x0 0x200001b71:0x7e:0x0 0x200001b71:0x84:0x0 0x200001b71:0xa:0x0 0x200001b71:0xe5:0x0 0x200001b71:0xea:0x0 0x200001b71:0xf0:0x0 0x200001b73:0xb60b:0x0 500 tnm /mnt/lustre/.lustre/.ltrash/MDT0000/0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000007:0x1:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd0:0x657:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbcc:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbd2:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbec:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc06:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc0c:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc12:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc1c:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc1f:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc25:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc28:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc2e:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc33:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc37:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc39:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc93:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc95:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3ee:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3f3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3f8:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x401:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x5:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0x10bc7:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0x1:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0x77:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0x79:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0x7e:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0x84:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0xa:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0xe5:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0xea:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0xf0:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b73:0xb60b:0x0: f118l.sanityn /mnt/lustre/.lustre/.ltrash/MDT0000/500: /mnt/lustre/.lustre/.ltrash/MDT0000/tnm: /mnt/lustre/.lustre/.ltrash/MDT0001: 0 0x240000404:0x5af:0x0 0x240000404:0x5b1:0x0 0x240000405:0x41:0x0 0x240000405:0xc:0x0 tnm /mnt/lustre/.lustre/.ltrash/MDT0001/0: /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000404:0x5af:0x0: /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000404:0x5b1:0x0: /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000405:0x41:0x0: /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000405:0xc:0x0: /mnt/lustre/.lustre/.ltrash/MDT0001/tnm: f118l.sanityn Phase 0/PR: remove dir 1:0x200000bd1:0xbd2:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xbd2:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xbd2:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xbd2:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xc28:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xc28:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xc28:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xc28:0x0 from Trash Can Phase 0/PR: remove dir 1:0x2000013a1:0x5:0x0 from Trash Can Phase 2/PO: remove dir 1:0x2000013a1:0x5:0x0 from Trash Can Skip stub dir 1:0x2000013a1:0x5:0x0 Phase 1/EN: remove dir 1:0x2000013a1:0x5:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xc12:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xc12:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xc12:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xc12:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xc06:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xc06:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xc06:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xc06:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200001b71:0xf0:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200001b71:0xf0:0x0 from Trash Can Skip stub dir 1:0x200001b71:0xf0:0x0 Phase 1/EN: remove dir 1:0x200001b71:0xf0:0x0 from Trash Can Phase 0/PR: remove dir 1:0 from Trash Can Phase 2/PO: remove dir 1:0 from Trash Can Phase 1/EN: remove dir 1:0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xc25:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xc25:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xc25:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xc25:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xbcc:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xbcc:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xbcc:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xbcc:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200001b71:0x84:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200001b71:0x84:0x0 from Trash Can Skip stub dir 1:0x200001b71:0x84:0x0 Phase 1/EN: remove dir 1:0x200001b71:0x84:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200001b71:0x7e:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200001b71:0x7e:0x0 from Trash Can Skip stub dir 1:0x200001b71:0x7e:0x0 Phase 1/EN: remove dir 1:0x200001b71:0x7e:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200001b71:0xea:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200001b71:0xea:0x0 from Trash Can Skip stub dir 1:0x200001b71:0xea:0x0 Phase 1/EN: remove dir 1:0x200001b71:0xea:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200001b71:0x77:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200001b71:0x77:0x0 from Trash Can Skip stub dir 1:0x200001b71:0x77:0x0 Phase 1/EN: remove dir 1:0x200001b71:0x77:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xbec:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xbec:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xbec:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xbec:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xc1f:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xc1f:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xc1f:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xc1f:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xc0c:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xc0c:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xc0c:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xc0c:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xbf3:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xbf3:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xbf3:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xbf3:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd0:0x657:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd0:0x657:0x0 from Trash Can Skip stub dir 1:0x200000bd0:0x657:0x0 Phase 1/EN: remove dir 1:0x200000bd0:0x657:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xc95:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xc95:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xc95:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xc95:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200001b71:0x1:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200001b71:0x1:0x0 from Trash Can Skip stub dir 1:0x200001b71:0x1:0x0 Phase 1/EN: remove dir 1:0x200001b71:0x1:0x0 from Trash Can Phase 0/PR: remove dir 1:0x2000013a1:0x3ee:0x0 from Trash Can Phase 2/PO: remove dir 1:0x2000013a1:0x3ee:0x0 from Trash Can Skip stub dir 1:0x2000013a1:0x3ee:0x0 Phase 1/EN: remove dir 1:0x2000013a1:0x3ee:0x0 from Trash Can Phase 0/PR: remove dir 1:0x2000013a1:0x3f3:0x0 from Trash Can Phase 2/PO: remove dir 1:0x2000013a1:0x3f3:0x0 from Trash Can Skip stub dir 1:0x2000013a1:0x3f3:0x0 Phase 1/EN: remove dir 1:0x2000013a1:0x3f3:0x0 from Trash Can Phase 0/PR: remove dir 1:0x2000013a1:0x401:0x0 from Trash Can Phase 2/PO: remove dir 1:0x2000013a1:0x401:0x0 from Trash Can Skip stub dir 1:0x2000013a1:0x401:0x0 Phase 1/EN: remove dir 1:0x2000013a1:0x401:0x0 from Trash Can Phase 0/PR: remove dir 1:500 from Trash Can Phase 2/PO: remove dir 1:500 from Trash Can Phase 1/EN: remove dir 1:500 from Trash Can Phase 0/PR: remove dir 1:0x200000007:0x1:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000007:0x1:0x0 from Trash Can Skip stub dir 1:0x200000007:0x1:0x0 Phase 1/EN: remove dir 1:0x200000007:0x1:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200001b71:0xa:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200001b71:0xa:0x0 from Trash Can Skip stub dir 1:0x200001b71:0xa:0x0 Phase 1/EN: remove dir 1:0x200001b71:0xa:0x0 from Trash Can Phase 0/PR: remove dir 1:0x2000013a1:0x3:0x0 from Trash Can Phase 2/PO: remove dir 1:0x2000013a1:0x3:0x0 from Trash Can Skip stub dir 1:0x2000013a1:0x3:0x0 Phase 1/EN: remove dir 1:0x2000013a1:0x3:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200001b71:0x10bc7:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200001b71:0x10bc7:0x0 from Trash Can Skip stub dir 1:0x200001b71:0x10bc7:0x0 Phase 1/EN: remove dir 1:0x200001b71:0x10bc7:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xc33:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xc33:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xc33:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xc33:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xc37:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xc37:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xc37:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xc37:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200001b73:0xb60b:0x0 from Trash Can Phase 1/EN: remove file 2:f118l.sanityn from Trash Can Phase 2/PO: remove dir 1:0x200001b73:0xb60b:0x0 from Trash Can Skip stub dir 1:0x200001b73:0xb60b:0x0 Phase 1/EN: remove dir 1:0x200001b73:0xb60b:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xc1c:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xc1c:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xc1c:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xc1c:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xc2e:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xc2e:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xc2e:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xc2e:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xc39:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xc39:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xc39:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xc39:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xc93:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xc93:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xc93:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xc93:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xbe7:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xbe7:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xbe7:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xbe7:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200000bd1:0xbe3:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200000bd1:0xbe3:0x0 from Trash Can Skip stub dir 1:0x200000bd1:0xbe3:0x0 Phase 1/EN: remove dir 1:0x200000bd1:0xbe3:0x0 from Trash Can Phase 0/PR: remove dir 1:0x200001b71:0x79:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200001b71:0x79:0x0 from Trash Can Skip stub dir 1:0x200001b71:0x79:0x0 Phase 1/EN: remove dir 1:0x200001b71:0x79:0x0 from Trash Can Phase 0/PR: remove dir 1:0x2000013a1:0x3f8:0x0 from Trash Can Phase 2/PO: remove dir 1:0x2000013a1:0x3f8:0x0 from Trash Can Skip stub dir 1:0x2000013a1:0x3f8:0x0 Phase 1/EN: remove dir 1:0x2000013a1:0x3f8:0x0 from Trash Can Phase 0/PR: remove dir 1:tnm from Trash Can Phase 2/PO: remove dir 1:tnm from Trash Can Phase 1/EN: remove dir 1:tnm from Trash Can Phase 0/PR: remove dir 1:0x200001b71:0xe5:0x0 from Trash Can Phase 2/PO: remove dir 1:0x200001b71:0xe5:0x0 from Trash Can Skip stub dir 1:0x200001b71:0xe5:0x0 Phase 1/EN: remove dir 1:0x200001b71:0xe5:0x0 from Trash Can Phase 0/PR: remove dir 1:0x240000405:0xc:0x0 from Trash Can Phase 2/PO: remove dir 1:0x240000405:0xc:0x0 from Trash Can Skip stub dir 1:0x240000405:0xc:0x0 Phase 1/EN: remove dir 1:0x240000405:0xc:0x0 from Trash Can Phase 0/PR: remove dir 1:0 from Trash Can Phase 2/PO: remove dir 1:0 from Trash Can Phase 1/EN: remove dir 1:0 from Trash Can Phase 0/PR: remove dir 1:tnm from Trash Can Phase 2/PO: remove dir 1:tnm from Trash Can Phase 1/EN: remove dir 1:tnm from Trash Can Phase 0/PR: remove dir 1:0x240000405:0x41:0x0 from Trash Can Phase 2/PO: remove dir 1:0x240000405:0x41:0x0 from Trash Can Skip stub dir 1:0x240000405:0x41:0x0 Phase 1/EN: remove dir 1:0x240000405:0x41:0x0 from Trash Can Phase 0/PR: remove dir 1:0x240000404:0x5af:0x0 from Trash Can Phase 2/PO: remove dir 1:0x240000404:0x5af:0x0 from Trash Can Skip stub dir 1:0x240000404:0x5af:0x0 Phase 1/EN: remove dir 1:0x240000404:0x5af:0x0 from Trash Can Phase 0/PR: remove dir 1:0x240000404:0x5b1:0x0 from Trash Can Phase 2/PO: remove dir 1:0x240000404:0x5b1:0x0 from Trash Can Skip stub dir 1:0x240000404:0x5b1:0x0 Phase 1/EN: remove dir 1:0x240000404:0x5b1:0x0 from Trash Can /mnt/lustre/.lustre/.ltrash: MDT0000 MDT0001 /mnt/lustre/.lustre/.ltrash/MDT0000: 0 0x200000007:0x1:0x0 0x200000bd0:0x657:0x0 0x200000bd1:0xbcc:0x0 0x200000bd1:0xbd2:0x0 0x200000bd1:0xbe3:0x0 0x200000bd1:0xbe7:0x0 0x200000bd1:0xbec:0x0 0x200000bd1:0xbf3:0x0 0x200000bd1:0xc06:0x0 0x200000bd1:0xc0c:0x0 0x200000bd1:0xc12:0x0 0x200000bd1:0xc1c:0x0 0x200000bd1:0xc1f:0x0 0x200000bd1:0xc25:0x0 0x200000bd1:0xc28:0x0 0x200000bd1:0xc2e:0x0 0x200000bd1:0xc33:0x0 0x200000bd1:0xc37:0x0 0x200000bd1:0xc39:0x0 0x200000bd1:0xc93:0x0 0x200000bd1:0xc95:0x0 0x2000013a1:0x3:0x0 0x2000013a1:0x3ee:0x0 0x2000013a1:0x3f3:0x0 0x2000013a1:0x3f8:0x0 0x2000013a1:0x401:0x0 0x2000013a1:0x5:0x0 0x200001b71:0x10bc7:0x0 0x200001b71:0x1:0x0 0x200001b71:0x77:0x0 0x200001b71:0x79:0x0 0x200001b71:0x7e:0x0 0x200001b71:0x84:0x0 0x200001b71:0xa:0x0 0x200001b71:0xe5:0x0 0x200001b71:0xea:0x0 0x200001b71:0xf0:0x0 0x200001b73:0xb60b:0x0 500 tnm /mnt/lustre/.lustre/.ltrash/MDT0000/0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000007:0x1:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd0:0x657:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbcc:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbd2:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbec:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc06:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc0c:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc12:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc1c:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc1f:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc25:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc28:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc2e:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc33:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc37:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc39:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc93:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000bd1:0xc95:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3ee:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3f3:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x3f8:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x401:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x2000013a1:0x5:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0x10bc7:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0x1:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0x77:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0x79:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0x7e:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0x84:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0xa:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0xe5:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0xea:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b71:0xf0:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200001b73:0xb60b:0x0: /mnt/lustre/.lustre/.ltrash/MDT0000/500: /mnt/lustre/.lustre/.ltrash/MDT0000/tnm: /mnt/lustre/.lustre/.ltrash/MDT0001: 0 0x240000404:0x5af:0x0 0x240000404:0x5b1:0x0 0x240000405:0x41:0x0 0x240000405:0xc:0x0 tnm /mnt/lustre/.lustre/.ltrash/MDT0001/0: /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000404:0x5af:0x0: /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000404:0x5b1:0x0: /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000405:0x41:0x0: /mnt/lustre/.lustre/.ltrash/MDT0001/0x240000405:0xc:0x0: /mnt/lustre/.lustre/.ltrash/MDT0001/tnm: 192.168.201.135@tcp:/lustre/d118l.sanityn /mnt/lustre_d118l.sanityn lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre_d118l.sanityn (opts:) mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 118l (10s) == sanityn test 118m: Access .ltrash under normal root dir and subdir mount point ========================================================== 00:14:47 (1779077687) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain /mnt/lustre/.ltrash: MDT0000 MDT0001 /mnt/lustre/.ltrash/MDT0000: 0 0x200000007:0x1:0x0 0x200000bd0:0x657:0x0 0x200000bd1:0xbcc:0x0 0x200000bd1:0xbd2:0x0 0x200000bd1:0xbe3:0x0 0x200000bd1:0xbe7:0x0 0x200000bd1:0xbec:0x0 0x200000bd1:0xbf3:0x0 0x200000bd1:0xc06:0x0 0x200000bd1:0xc0c:0x0 0x200000bd1:0xc12:0x0 0x200000bd1:0xc1c:0x0 0x200000bd1:0xc1f:0x0 0x200000bd1:0xc25:0x0 0x200000bd1:0xc28:0x0 0x200000bd1:0xc2e:0x0 0x200000bd1:0xc33:0x0 0x200000bd1:0xc37:0x0 0x200000bd1:0xc39:0x0 0x200000bd1:0xc93:0x0 0x200000bd1:0xc95:0x0 0x2000013a1:0x3:0x0 0x2000013a1:0x3ee:0x0 0x2000013a1:0x3f3:0x0 0x2000013a1:0x3f8:0x0 0x2000013a1:0x401:0x0 0x2000013a1:0x5:0x0 0x200001b71:0x10bc7:0x0 0x200001b71:0x1:0x0 0x200001b71:0x77:0x0 0x200001b71:0x79:0x0 0x200001b71:0x7e:0x0 0x200001b71:0x84:0x0 0x200001b71:0xa:0x0 0x200001b71:0xe5:0x0 0x200001b71:0xea:0x0 0x200001b71:0xf0:0x0 0x200001b73:0xb60b:0x0 500 tnm /mnt/lustre/.ltrash/MDT0000/0: /mnt/lustre/.ltrash/MDT0000/0x200000007:0x1:0x0: /mnt/lustre/.ltrash/MDT0000/0x200000bd0:0x657:0x0: /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xbcc:0x0: /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xbd2:0x0: /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xbe3:0x0: /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0: /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xbec:0x0: /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0: /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc06:0x0: /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc0c:0x0: /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc12:0x0: /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc1c:0x0: /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc1f:0x0: /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc25:0x0: /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc28:0x0: /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc2e:0x0: /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc33:0x0: /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc37:0x0: /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc39:0x0: /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc93:0x0: /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc95:0x0: /mnt/lustre/.ltrash/MDT0000/0x2000013a1:0x3:0x0: /mnt/lustre/.ltrash/MDT0000/0x2000013a1:0x3ee:0x0: /mnt/lustre/.ltrash/MDT0000/0x2000013a1:0x3f3:0x0: /mnt/lustre/.ltrash/MDT0000/0x2000013a1:0x3f8:0x0: /mnt/lustre/.ltrash/MDT0000/0x2000013a1:0x401:0x0: /mnt/lustre/.ltrash/MDT0000/0x2000013a1:0x5:0x0: /mnt/lustre/.ltrash/MDT0000/0x200001b71:0x10bc7:0x0: /mnt/lustre/.ltrash/MDT0000/0x200001b71:0x1:0x0: /mnt/lustre/.ltrash/MDT0000/0x200001b71:0x77:0x0: /mnt/lustre/.ltrash/MDT0000/0x200001b71:0x79:0x0: /mnt/lustre/.ltrash/MDT0000/0x200001b71:0x7e:0x0: /mnt/lustre/.ltrash/MDT0000/0x200001b71:0x84:0x0: /mnt/lustre/.ltrash/MDT0000/0x200001b71:0xa:0x0: /mnt/lustre/.ltrash/MDT0000/0x200001b71:0xe5:0x0: /mnt/lustre/.ltrash/MDT0000/0x200001b71:0xea:0x0: /mnt/lustre/.ltrash/MDT0000/0x200001b71:0xf0:0x0: /mnt/lustre/.ltrash/MDT0000/0x200001b73:0xb60b:0x0: /mnt/lustre/.ltrash/MDT0000/500: /mnt/lustre/.ltrash/MDT0000/tnm: /mnt/lustre/.ltrash/MDT0001: 0 0x240000404:0x5af:0x0 0x240000404:0x5b1:0x0 0x240000405:0x41:0x0 0x240000405:0xc:0x0 tnm /mnt/lustre/.ltrash/MDT0001/0: /mnt/lustre/.ltrash/MDT0001/0x240000404:0x5af:0x0: /mnt/lustre/.ltrash/MDT0001/0x240000404:0x5b1:0x0: /mnt/lustre/.ltrash/MDT0001/0x240000405:0x41:0x0: /mnt/lustre/.ltrash/MDT0001/0x240000405:0xc:0x0: /mnt/lustre/.ltrash/MDT0001/tnm: Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre/d118m.sanityn /mnt/lustre_d118m.sanityn /mnt/lustre_d118m.sanityn/.ltrash: MDT0000 MDT0001 /mnt/lustre_d118m.sanityn/.ltrash/MDT0000: 0 0x200000007:0x1:0x0 0x200000bd0:0x657:0x0 0x200000bd1:0xbcc:0x0 0x200000bd1:0xbd2:0x0 0x200000bd1:0xbe3:0x0 0x200000bd1:0xbe7:0x0 0x200000bd1:0xbec:0x0 0x200000bd1:0xbf3:0x0 0x200000bd1:0xc06:0x0 0x200000bd1:0xc0c:0x0 0x200000bd1:0xc12:0x0 0x200000bd1:0xc1c:0x0 0x200000bd1:0xc1f:0x0 0x200000bd1:0xc25:0x0 0x200000bd1:0xc28:0x0 0x200000bd1:0xc2e:0x0 0x200000bd1:0xc33:0x0 0x200000bd1:0xc37:0x0 0x200000bd1:0xc39:0x0 0x200000bd1:0xc93:0x0 0x200000bd1:0xc95:0x0 0x2000013a1:0x3:0x0 0x2000013a1:0x3ee:0x0 0x2000013a1:0x3f3:0x0 0x2000013a1:0x3f8:0x0 0x2000013a1:0x401:0x0 0x2000013a1:0x5:0x0 0x200001b71:0x10bc7:0x0 0x200001b71:0x1:0x0 0x200001b71:0x77:0x0 0x200001b71:0x79:0x0 0x200001b71:0x7e:0x0 0x200001b71:0x84:0x0 0x200001b71:0xa:0x0 0x200001b71:0xe5:0x0 0x200001b71:0xea:0x0 0x200001b71:0xf0:0x0 0x200001b73:0xb60b:0x0 0x200001b73:0xb60c:0x0 500 tnm /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200000007:0x1:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200000bd0:0x657:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200000bd1:0xbcc:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200000bd1:0xbd2:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200000bd1:0xbe3:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200000bd1:0xbec:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200000bd1:0xc06:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200000bd1:0xc0c:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200000bd1:0xc12:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200000bd1:0xc1c:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200000bd1:0xc1f:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200000bd1:0xc25:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200000bd1:0xc28:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200000bd1:0xc2e:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200000bd1:0xc33:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200000bd1:0xc37:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200000bd1:0xc39:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200000bd1:0xc93:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200000bd1:0xc95:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x2000013a1:0x3:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x2000013a1:0x3ee:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x2000013a1:0x3f3:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x2000013a1:0x3f8:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x2000013a1:0x401:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x2000013a1:0x5:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200001b71:0x10bc7:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200001b71:0x1:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200001b71:0x77:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200001b71:0x79:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200001b71:0x7e:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200001b71:0x84:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200001b71:0xa:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200001b71:0xe5:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200001b71:0xea:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200001b71:0xf0:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200001b73:0xb60b:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200001b73:0xb60c:0x0: tdir1 tfile2 /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200001b73:0xb60c:0x0/tdir1: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/500: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/tnm: /mnt/lustre_d118m.sanityn/.ltrash/MDT0001: 0 0x240000404:0x5af:0x0 0x240000404:0x5b1:0x0 0x240000405:0x41:0x0 0x240000405:0xc:0x0 tnm /mnt/lustre_d118m.sanityn/.ltrash/MDT0001/0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0001/0x240000404:0x5af:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0001/0x240000404:0x5b1:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0001/0x240000405:0x41:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0001/0x240000405:0xc:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0001/tnm: /mnt/lustre_d118m.sanityn/.ltrash: MDT0000 MDT0001 /mnt/lustre_d118m.sanityn/.ltrash/MDT0000: 0 0x200000007:0x1:0x0 0x200000bd0:0x657:0x0 0x200000bd1:0xbcc:0x0 0x200000bd1:0xbd2:0x0 0x200000bd1:0xbe3:0x0 0x200000bd1:0xbe7:0x0 0x200000bd1:0xbec:0x0 0x200000bd1:0xbf3:0x0 0x200000bd1:0xc06:0x0 0x200000bd1:0xc0c:0x0 0x200000bd1:0xc12:0x0 0x200000bd1:0xc1c:0x0 0x200000bd1:0xc1f:0x0 0x200000bd1:0xc25:0x0 0x200000bd1:0xc28:0x0 0x200000bd1:0xc2e:0x0 0x200000bd1:0xc33:0x0 0x200000bd1:0xc37:0x0 0x200000bd1:0xc39:0x0 0x200000bd1:0xc93:0x0 0x200000bd1:0xc95:0x0 0x2000013a1:0x3:0x0 0x2000013a1:0x3ee:0x0 0x2000013a1:0x3f3:0x0 0x2000013a1:0x3f8:0x0 0x2000013a1:0x401:0x0 0x2000013a1:0x5:0x0 0x200001b71:0x10bc7:0x0 0x200001b71:0x1:0x0 0x200001b71:0x77:0x0 0x200001b71:0x79:0x0 0x200001b71:0x7e:0x0 0x200001b71:0x84:0x0 0x200001b71:0xa:0x0 0x200001b71:0xe5:0x0 0x200001b71:0xea:0x0 0x200001b71:0xf0:0x0 0x200001b73:0xb60b:0x0 0x200001b73:0xb60c:0x0 500 tnm /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200000007:0x1:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200000bd0:0x657:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200000bd1:0xbcc:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200000bd1:0xbd2:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200000bd1:0xbe3:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200000bd1:0xbec:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200000bd1:0xc06:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200000bd1:0xc0c:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200000bd1:0xc12:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200000bd1:0xc1c:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200000bd1:0xc1f:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200000bd1:0xc25:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200000bd1:0xc28:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200000bd1:0xc2e:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200000bd1:0xc33:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200000bd1:0xc37:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200000bd1:0xc39:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200000bd1:0xc93:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200000bd1:0xc95:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x2000013a1:0x3:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x2000013a1:0x3ee:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x2000013a1:0x3f3:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x2000013a1:0x3f8:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x2000013a1:0x401:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x2000013a1:0x5:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200001b71:0x10bc7:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200001b71:0x1:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200001b71:0x77:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200001b71:0x79:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200001b71:0x7e:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200001b71:0x84:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200001b71:0xa:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200001b71:0xe5:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200001b71:0xea:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200001b71:0xf0:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200001b73:0xb60b:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200001b73:0xb60c:0x0: tdir1 tdir3 tfile2 tfile4 /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200001b73:0xb60c:0x0/tdir1: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/0x200001b73:0xb60c:0x0/tdir3: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/500: /mnt/lustre_d118m.sanityn/.ltrash/MDT0000/tnm: /mnt/lustre_d118m.sanityn/.ltrash/MDT0001: 0 0x240000404:0x5af:0x0 0x240000404:0x5b1:0x0 0x240000405:0x41:0x0 0x240000405:0xc:0x0 tnm /mnt/lustre_d118m.sanityn/.ltrash/MDT0001/0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0001/0x240000404:0x5af:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0001/0x240000404:0x5b1:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0001/0x240000405:0x41:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0001/0x240000405:0xc:0x0: /mnt/lustre_d118m.sanityn/.ltrash/MDT0001/tnm: 192.168.201.135@tcp:/lustre/d118m.sanityn /mnt/lustre_d118m.sanityn lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre_d118m.sanityn (opts:) mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 118m (39s) == sanityn test 118n: lfs trash state should return state for a symlink file correctly ========================================================== 00:15:26 (1779077726) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain file: /mnt/lustre/f118n.sanityn.lnk, flags: 0 file: /mnt/lustre/.Trash/f118n.sanityn, flags: 0X1DCFBF5600080002 unrm, uid: 0 gid: 0 projid: 0 timestamp: 1779077729 file: /mnt/lustre/.Trash/f118n.sanityn.lnk, flags: 0X2 unrm, uid: 0 gid: 0 projid: 0 timestamp: 1779077729 mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 118n (8s) == sanityn test 118o: lfs trash unrm should work with relative path ========================================================== 00:15:34 (1779077734) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain /mnt/lustre ~ .: d100b.sanityn d100c.sanityn d100d.sanityn d100e.sanityn d102.sanityn.remote d102.sanityn.stripe d103.sanityn d105.sanityn d11 d110.sanityn d112.sanityn d114.sanityn d115.sanityn d116.sanityn d117b.sanityn d117c.sanityn d117d.sanityn d117e.sanityn d117f.sanityn d117g.sanityn d117h.sanityn d117i.sanityn d117j.sanityn d117k.sanityn d117l.sanityn d117m.sanityn d117n.sanityn d117o.sanityn d117pb.sanityn d117p.sanityn d117q.sanityn d117r.sanityn d117s.sanityn d117t.sanityn d117ub.sanityn d117u.sanityn d117w.sanityn d117x.sanityn d117ya.sanityn d117yb.sanityn d117yc.sanityn d117za.sanityn d117zb.sanityn d117zc.sanityn d117zd.sanityn d118Aa.sanityn d118Ab.sanityn d118Ba.sanityn d118Bb.sanityn d118C.sanityn d118d.sanityn d118f.sanityn d118g.sanityn d118j.sanityn d118k.sanityn d118l.sanityn d118m.sanityn d118o.sanityn d13 d14aa.sanityn d14ab.sanityn d14b.sanityn d14c.sanityn d14d.sanityn d16h.sanityn d16i.sanityn d20.sanityn d30.sanityn d31a.sanityn d31b.sanityn d31s.sanityn d33c.sanityn d35.sanityn d36.sanityn d37.sanityn d40d.sanityn d43j.sanityn d60.sanityn d70a.sanityn d70b.sanityn d71b.sanityn d71c.sanityn d77m.sanityn d79.sanityn d80b.sanityn d81a.sanityn d81b.sanityn d90.sanityn d91.sanityn f10 f101b.sanityn f104.sanityn f106a.sanityn f106b.sanityn f106c.sanityn f107a.sanityn f107b.sanityn f108a.sanityn f10b f110.sanityn f118o.sanityn f16a.sanityn f16a.sanityn.fsxgood f16a.sanityn.fsxlog f16b.sanityn f16b.sanityn.fsxgood f16b.sanityn.fsxlog f16c.sanityn.fsxgood f16c.sanityn.fsxlog f17.sanityn f17.sanityn-1 f17.sanityn-2 f24a.sanityn f24b.sanityn f26a f26b.sanityn f26c.sanityn f2a f2b f2e.sanityn f2g.sanityn f31r.sanityn.target f31t.sanityn f31t.sanityn.old f34.sanityn f39a.sanityn-1 f39b.sanityn f39c.sanityn f39d.sanityn f3.sanityn f4 f50.sanityn f51d.sanityn f51e.sanityn f71a.sanityn f71d.sanityn f72.sanityn-2 f74.sanityn-1 f74.sanityn-2 f75.sanityn f9 f93.sanityn-1 f93.sanityn-2 f94.sanityn f95a.sanityn f95b.sanityn f9b.sanityn fsxfile.1 fsxfile.1.fsxgood fsxfile.1.fsxlog fsxfile.2 fsxfile.2.fsxgood fsxfile.2.fsxlog fsxfile.3 fsxfile.3.fsxgood fsxfile.3.fsxlog fsxfile.4 fsxfile.4.fsxgood fsxfile.4.fsxlog fsxfile.5 fsxfile.5.fsxgood fsxfile.5.fsxlog md_stats tfile.2 ./d100b.sanityn: dom ./d100c.sanityn: dom ./d100d.sanityn: dom ./d100e.sanityn: dom ./d102.sanityn.remote: subdir ./d102.sanityn.remote/subdir: ./d102.sanityn.stripe: subdir ./d102.sanityn.stripe/subdir: ./d103.sanityn: f103.sanityn ./d105.sanityn: f105.sanityn ./d11: f ./d110.sanityn: f1 f2 ./d112.sanityn: s1 ./d112.sanityn/s1: s2 ./d112.sanityn/s1/s2: s3 ./d112.sanityn/s1/s2/s3: d1 d10 d11 d12 d13 d14 d15 d16 d17 d18 d19 d2 d20 d21 d22 d23 d24 d25 d26 d27 d28 d29 d3 d30 d31 d32 d33 d34 d35 d36 d37 d38 d39 d4 d40 d41 d42 d43 d44 d45 d46 d47 d48 d49 d5 d50 d51 d52 d53 d54 d55 d56 d57 d58 d59 d6 d60 d61 d62 d63 d64 d7 d8 d9 s1 s10 s11 s12 s13 s14 s15 s16 s17 s18 s19 s2 s20 s21 s22 s23 s24 s25 s26 s27 s28 s29 s3 s30 s31 s32 s33 s34 s35 s36 s37 s38 s39 s4 s40 s41 s42 s43 s44 s45 s46 s47 s48 s49 s5 s50 s51 s52 s53 s54 s55 s56 s57 s58 s59 s6 s60 s61 s62 s63 s64 s7 s8 s9 ./d112.sanityn/s1/s2/s3/d1: ./d112.sanityn/s1/s2/s3/d10: ./d112.sanityn/s1/s2/s3/d11: ./d112.sanityn/s1/s2/s3/d12: ./d112.sanityn/s1/s2/s3/d13: ./d112.sanityn/s1/s2/s3/d14: ./d112.sanityn/s1/s2/s3/d15: ./d112.sanityn/s1/s2/s3/d16: ./d112.sanityn/s1/s2/s3/d17: ./d112.sanityn/s1/s2/s3/d18: ./d112.sanityn/s1/s2/s3/d19: ./d112.sanityn/s1/s2/s3/d2: ./d112.sanityn/s1/s2/s3/d20: ./d112.sanityn/s1/s2/s3/d21: ./d112.sanityn/s1/s2/s3/d22: ./d112.sanityn/s1/s2/s3/d23: ./d112.sanityn/s1/s2/s3/d24: ./d112.sanityn/s1/s2/s3/d25: ./d112.sanityn/s1/s2/s3/d26: ./d112.sanityn/s1/s2/s3/d27: ./d112.sanityn/s1/s2/s3/d28: ./d112.sanityn/s1/s2/s3/d29: ./d112.sanityn/s1/s2/s3/d3: ./d112.sanityn/s1/s2/s3/d30: ./d112.sanityn/s1/s2/s3/d31: ./d112.sanityn/s1/s2/s3/d32: ./d112.sanityn/s1/s2/s3/d33: ./d112.sanityn/s1/s2/s3/d34: ./d112.sanityn/s1/s2/s3/d35: ./d112.sanityn/s1/s2/s3/d36: ./d112.sanityn/s1/s2/s3/d37: ./d112.sanityn/s1/s2/s3/d38: ./d112.sanityn/s1/s2/s3/d39: ./d112.sanityn/s1/s2/s3/d4: ./d112.sanityn/s1/s2/s3/d40: ./d112.sanityn/s1/s2/s3/d41: ./d112.sanityn/s1/s2/s3/d42: ./d112.sanityn/s1/s2/s3/d43: ./d112.sanityn/s1/s2/s3/d44: ./d112.sanityn/s1/s2/s3/d45: ./d112.sanityn/s1/s2/s3/d46: ./d112.sanityn/s1/s2/s3/d47: ./d112.sanityn/s1/s2/s3/d48: ./d112.sanityn/s1/s2/s3/d49: ./d112.sanityn/s1/s2/s3/d5: ./d112.sanityn/s1/s2/s3/d50: ./d112.sanityn/s1/s2/s3/d51: ./d112.sanityn/s1/s2/s3/d52: ./d112.sanityn/s1/s2/s3/d53: ./d112.sanityn/s1/s2/s3/d54: ./d112.sanityn/s1/s2/s3/d55: ./d112.sanityn/s1/s2/s3/d56: ./d112.sanityn/s1/s2/s3/d57: ./d112.sanityn/s1/s2/s3/d58: ./d112.sanityn/s1/s2/s3/d59: ./d112.sanityn/s1/s2/s3/d6: ./d112.sanityn/s1/s2/s3/d60: ./d112.sanityn/s1/s2/s3/d61: ./d112.sanityn/s1/s2/s3/d62: ./d112.sanityn/s1/s2/s3/d63: ./d112.sanityn/s1/s2/s3/d64: ./d112.sanityn/s1/s2/s3/d7: ./d112.sanityn/s1/s2/s3/d8: ./d112.sanityn/s1/s2/s3/d9: ./d112.sanityn/s1/s2/s3/s1: ./d112.sanityn/s1/s2/s3/s10: ./d112.sanityn/s1/s2/s3/s11: ./d112.sanityn/s1/s2/s3/s12: ./d112.sanityn/s1/s2/s3/s13: ./d112.sanityn/s1/s2/s3/s14: ./d112.sanityn/s1/s2/s3/s15: ./d112.sanityn/s1/s2/s3/s16: ./d112.sanityn/s1/s2/s3/s17: ./d112.sanityn/s1/s2/s3/s18: ./d112.sanityn/s1/s2/s3/s19: ./d112.sanityn/s1/s2/s3/s2: ./d112.sanityn/s1/s2/s3/s20: ./d112.sanityn/s1/s2/s3/s21: ./d112.sanityn/s1/s2/s3/s22: ./d112.sanityn/s1/s2/s3/s23: ./d112.sanityn/s1/s2/s3/s24: ./d112.sanityn/s1/s2/s3/s25: ./d112.sanityn/s1/s2/s3/s26: ./d112.sanityn/s1/s2/s3/s27: ./d112.sanityn/s1/s2/s3/s28: ./d112.sanityn/s1/s2/s3/s29: ./d112.sanityn/s1/s2/s3/s3: ./d112.sanityn/s1/s2/s3/s30: ./d112.sanityn/s1/s2/s3/s31: ./d112.sanityn/s1/s2/s3/s32: ./d112.sanityn/s1/s2/s3/s33: ./d112.sanityn/s1/s2/s3/s34: ./d112.sanityn/s1/s2/s3/s35: ./d112.sanityn/s1/s2/s3/s36: ./d112.sanityn/s1/s2/s3/s37: ./d112.sanityn/s1/s2/s3/s38: ./d112.sanityn/s1/s2/s3/s39: ./d112.sanityn/s1/s2/s3/s4: ./d112.sanityn/s1/s2/s3/s40: ./d112.sanityn/s1/s2/s3/s41: ./d112.sanityn/s1/s2/s3/s42: ./d112.sanityn/s1/s2/s3/s43: ./d112.sanityn/s1/s2/s3/s44: ./d112.sanityn/s1/s2/s3/s45: ./d112.sanityn/s1/s2/s3/s46: ./d112.sanityn/s1/s2/s3/s47: ./d112.sanityn/s1/s2/s3/s48: ./d112.sanityn/s1/s2/s3/s49: ./d112.sanityn/s1/s2/s3/s5: ./d112.sanityn/s1/s2/s3/s50: ./d112.sanityn/s1/s2/s3/s51: ./d112.sanityn/s1/s2/s3/s52: ./d112.sanityn/s1/s2/s3/s53: ./d112.sanityn/s1/s2/s3/s54: ./d112.sanityn/s1/s2/s3/s55: ./d112.sanityn/s1/s2/s3/s56: ./d112.sanityn/s1/s2/s3/s57: ./d112.sanityn/s1/s2/s3/s58: ./d112.sanityn/s1/s2/s3/s59: ./d112.sanityn/s1/s2/s3/s6: ./d112.sanityn/s1/s2/s3/s60: ./d112.sanityn/s1/s2/s3/s61: ./d112.sanityn/s1/s2/s3/s62: ./d112.sanityn/s1/s2/s3/s63: ./d112.sanityn/s1/s2/s3/s64: ./d112.sanityn/s1/s2/s3/s7: ./d112.sanityn/s1/s2/s3/s8: ./d112.sanityn/s1/s2/s3/s9: ./d114.sanityn: l1 ./d114.sanityn/l1: l2 ./d114.sanityn/l1/l2: l3 ./d114.sanityn/l1/l2/l3: ./d115.sanityn: f0 f1 f10 f100 f1000 f1001 f1002 f1003 f1004 f1005 f1006 f1007 f1008 f1009 f101 f1010 f1011 f1012 f1013 f1014 f1015 f1016 f1017 f1018 f1019 f102 f1020 f1021 f1022 f1023 f1024 f1025 f1026 f1027 f1028 f1029 f103 f1030 f1031 f1032 f1033 f1034 f1035 f1036 f1037 f1038 f1039 f104 f1040 f1041 f1042 f1043 f1044 f1045 f1046 f1047 f1048 f1049 f105 f1050 f1051 f1052 f1053 f1054 f1055 f1056 f1057 f1058 f1059 f106 f1060 f1061 f1062 f1063 f1064 f1065 f1066 f1067 f1068 f1069 f107 f1070 f1071 f1072 f1073 f1074 f1075 f1076 f1077 f1078 f1079 f108 f1080 f1081 f1082 f1083 f1084 f1085 f1086 f1087 f1088 f1089 f109 f1090 f1091 f1092 f1093 f1094 f1095 f1096 f1097 f1098 f1099 f11 f110 f1100 f1101 f1102 f1103 f1104 f1105 f1106 f1107 f1108 f1109 f111 f1110 f1111 f1112 f1113 f1114 f1115 f1116 f1117 f1118 f1119 f112 f1120 f1121 f1122 f1123 f1124 f1125 f1126 f1127 f1128 f1129 f113 f1130 f1131 f1132 f1133 f1134 f1135 f1136 f1137 f1138 f1139 f114 f1140 f1141 f1142 f1143 f1144 f1145 f1146 f1147 f1148 f1149 f115 f1150 f1151 f1152 f1153 f1154 f1155 f1156 f1157 f1158 f1159 f116 f1160 f1161 f1162 f1163 f1164 f1165 f1166 f1167 f1168 f1169 f117 f1170 f1171 f1172 f1173 f1174 f1175 f1176 f1177 f1178 f1179 f118 f1180 f1181 f1182 f1183 f1184 f1185 f1186 f1187 f1188 f1189 f119 f1190 f1191 f1192 f1193 f1194 f1195 f1196 f1197 f1198 f1199 f12 f120 f1200 f1201 f1202 f1203 f1204 f1205 f1206 f1207 f1208 f1209 f121 f1210 f1211 f1212 f1213 f1214 f1215 f1216 f1217 f1218 f1219 f122 f1220 f1221 f1222 f1223 f1224 f1225 f1226 f1227 f1228 f1229 f123 f1230 f1231 f1232 f1233 f1234 f1235 f1236 f1237 f1238 f1239 f124 f1240 f1241 f1242 f1243 f1244 f1245 f1246 f1247 f1248 f1249 f125 f1250 f1251 f1252 f1253 f1254 f1255 f1256 f1257 f1258 f1259 f126 f1260 f1261 f1262 f1263 f1264 f1265 f1266 f1267 f1268 f1269 f127 f1270 f1271 f1272 f1273 f1274 f1275 f1276 f1277 f1278 f1279 f128 f1280 f1281 f1282 f1283 f1284 f1285 f1286 f1287 f1288 f1289 f129 f1290 f1291 f1292 f1293 f1294 f1295 f1296 f1297 f1298 f1299 f13 f130 f1300 f1301 f1302 f1303 f1304 f1305 f1306 f1307 f1308 f1309 f131 f1310 f1311 f1312 f1313 f1314 f1315 f1316 f1317 f1318 f1319 f132 f1320 f1321 f1322 f1323 f1324 f1325 f1326 f1327 f1328 f1329 f133 f1330 f1331 f1332 f1333 f1334 f1335 f1336 f1337 f1338 f1339 f134 f1340 f1341 f1342 f1343 f1344 f1345 f1346 f1347 f1348 f1349 f135 f1350 f1351 f1352 f1353 f1354 f1355 f1356 f1357 f1358 f1359 f136 f1360 f1361 f1362 f1363 f1364 f1365 f1366 f1367 f1368 f1369 f137 f1370 f1371 f1372 f1373 f1374 f1375 f1376 f1377 f1378 f1379 f138 f1380 f1381 f1382 f1383 f1384 f1385 f1386 f1387 f1388 f1389 f139 f1390 f1391 f1392 f1393 f1394 f1395 f1396 f1397 f1398 f1399 f14 f140 f1400 f1401 f1402 f1403 f1404 f1405 f1406 f1407 f1408 f1409 f141 f1410 f1411 f1412 f1413 f1414 f1415 f1416 f1417 f1418 f1419 f142 f1420 f1421 f1422 f1423 f1424 f1425 f1426 f1427 f1428 f1429 f143 f1430 f1431 f1432 f1433 f1434 f1435 f1436 f1437 f1438 f1439 f144 f1440 f1441 f1442 f1443 f1444 f1445 f1446 f1447 f1448 f1449 f145 f1450 f1451 f1452 f1453 f1454 f1455 f1456 f1457 f1458 f1459 f146 f1460 f1461 f1462 f1463 f1464 f1465 f1466 f1467 f1468 f1469 f147 f1470 f1471 f1472 f1473 f1474 f1475 f1476 f1477 f1478 f1479 f148 f1480 f1481 f1482 f1483 f1484 f1485 f1486 f1487 f1488 f1489 f149 f1490 f1491 f1492 f1493 f1494 f1495 f1496 f1497 f1498 f1499 f15 f150 f1500 f1501 f1502 f1503 f1504 f1505 f1506 f1507 f1508 f1509 f151 f1510 f1511 f1512 f1513 f1514 f1515 f1516 f1517 f1518 f1519 f152 f1520 f1521 f1522 f1523 f1524 f1525 f1526 f1527 f1528 f1529 f153 f1530 f1531 f1532 f1533 f1534 f1535 f1536 f1537 f1538 f1539 f154 f1540 f1541 f1542 f1543 f1544 f1545 f1546 f1547 f1548 f1549 f155 f1550 f1551 f1552 f1553 f1554 f1555 f1556 f1557 f1558 f1559 f156 f1560 f1561 f1562 f1563 f1564 f1565 f1566 f1567 f1568 f1569 f157 f1570 f1571 f1572 f1573 f1574 f1575 f1576 f1577 f1578 f1579 f158 f1580 f1581 f1582 f1583 f1584 f1585 f1586 f1587 f1588 f1589 f159 f1590 f1591 f1592 f1593 f1594 f1595 f1596 f1597 f1598 f1599 f16 f160 f1600 f1601 f1602 f1603 f1604 f1605 f1606 f1607 f1608 f1609 f161 f1610 f1611 f1612 f1613 f1614 f1615 f1616 f1617 f1618 f1619 f162 f1620 f1621 f1622 f1623 f1624 f1625 f1626 f1627 f1628 f1629 f163 f1630 f1631 f1632 f1633 f1634 f1635 f1636 f1637 f1638 f1639 f164 f1640 f1641 f1642 f1643 f1644 f1645 f1646 f1647 f1648 f1649 f165 f1650 f1651 f1652 f1653 f1654 f1655 f1656 f1657 f1658 f1659 f166 f1660 f1661 f1662 f1663 f1664 f1665 f1666 f1667 f1668 f1669 f167 f1670 f1671 f1672 f1673 f1674 f1675 f1676 f1677 f1678 f1679 f168 f1680 f1681 f1682 f1683 f1684 f1685 f1686 f1687 f1688 f1689 f169 f1690 f1691 f1692 f1693 f1694 f1695 f1696 f1697 f1698 f1699 f17 f170 f1700 f1701 f1702 f1703 f1704 f1705 f1706 f1707 f1708 f1709 f171 f1710 f1711 f1712 f1713 f1714 f1715 f1716 f1717 f1718 f1719 f172 f1720 f1721 f1722 f1723 f1724 f1725 f1726 f1727 f1728 f1729 f173 f1730 f1731 f1732 f1733 f1734 f1735 f1736 f1737 f1738 f1739 f174 f1740 f1741 f1742 f1743 f1744 f1745 f1746 f1747 f1748 f1749 f175 f1750 f1751 f1752 f1753 f1754 f1755 f1756 f1757 f1758 f1759 f176 f1760 f1761 f1762 f1763 f1764 f1765 f1766 f1767 f1768 f1769 f177 f1770 f1771 f1772 f1773 f1774 f1775 f1776 f1777 f1778 f1779 f178 f1780 f1781 f1782 f1783 f1784 f1785 f1786 f1787 f1788 f1789 f179 f1790 f1791 f1792 f1793 f1794 f1795 f1796 f1797 f1798 f1799 f18 f180 f1800 f1801 f1802 f1803 f1804 f1805 f1806 f1807 f1808 f1809 f181 f1810 f1811 f1812 f1813 f1814 f1815 f1816 f1817 f1818 f1819 f182 f1820 f1821 f1822 f1823 f1824 f1825 f1826 f1827 f1828 f1829 f183 f1830 f1831 f1832 f1833 f1834 f1835 f1836 f1837 f1838 f1839 f184 f1840 f1841 f1842 f1843 f1844 f1845 f1846 f1847 f1848 f1849 f185 f1850 f1851 f1852 f1853 f1854 f1855 f1856 f1857 f1858 f1859 f186 f1860 f1861 f1862 f1863 f1864 f1865 f1866 f1867 f1868 f1869 f187 f1870 f1871 f1872 f1873 f1874 f1875 f1876 f1877 f1878 f1879 f188 f1880 f1881 f1882 f1883 f1884 f1885 f1886 f1887 f1888 f1889 f189 f1890 f1891 f1892 f1893 f1894 f1895 f1896 f1897 f1898 f1899 f19 f190 f1900 f1901 f1902 f1903 f1904 f1905 f1906 f1907 f1908 f1909 f191 f1910 f1911 f1912 f1913 f1914 f1915 f1916 f1917 f1918 f1919 f192 f1920 f1921 f1922 f1923 f1924 f1925 f1926 f1927 f1928 f1929 f193 f1930 f1931 f1932 f1933 f1934 f1935 f1936 f1937 f1938 f1939 f194 f1940 f1941 f1942 f1943 f1944 f1945 f1946 f1947 f1948 f1949 f195 f1950 f1951 f1952 f1953 f1954 f1955 f1956 f1957 f1958 f1959 f196 f1960 f1961 f1962 f1963 f1964 f1965 f1966 f1967 f1968 f1969 f197 f1970 f1971 f1972 f1973 f1974 f1975 f1976 f1977 f1978 f1979 f198 f1980 f1981 f1982 f1983 f1984 f1985 f1986 f1987 f1988 f1989 f199 f1990 f1991 f1992 f1993 f1994 f1995 f1996 f1997 f1998 f1999 f2 f20 f200 f2000 f2001 f2002 f2003 f2004 f2005 f2006 f2007 f2008 f2009 f201 f2010 f2011 f2012 f2013 f2014 f2015 f2016 f2017 f2018 f2019 f202 f2020 f2021 f2022 f2023 f2024 f2025 f2026 f2027 f2028 f2029 f203 f2030 f2031 f2032 f2033 f2034 f2035 f2036 f2037 f2038 f2039 f204 f2040 f2041 f2042 f2043 f2044 f2045 f2046 f2047 f2048 f2049 f205 f2050 f2051 f2052 f2053 f2054 f2055 f2056 f2057 f2058 f2059 f206 f2060 f2061 f2062 f2063 f2064 f2065 f2066 f2067 f2068 f2069 f207 f2070 f2071 f2072 f2073 f2074 f2075 f2076 f2077 f2078 f2079 f208 f2080 f2081 f2082 f2083 f2084 f2085 f2086 f2087 f2088 f2089 f209 f2090 f2091 f2092 f2093 f2094 f2095 f2096 f2097 f2098 f2099 f21 f210 f2100 f2101 f2102 f2103 f2104 f2105 f2106 f2107 f2108 f2109 f211 f2110 f2111 f2112 f2113 f2114 f2115 f2116 f2117 f2118 f2119 f212 f2120 f2121 f2122 f2123 f2124 f2125 f2126 f2127 f2128 f2129 f213 f2130 f2131 f2132 f2133 f2134 f2135 f2136 f2137 f2138 f2139 f214 f2140 f2141 f2142 f2143 f2144 f2145 f2146 f2147 f2148 f2149 f215 f2150 f2151 f2152 f2153 f2154 f2155 f2156 f2157 f2158 f2159 f216 f2160 f2161 f2162 f2163 f2164 f2165 f2166 f2167 f2168 f2169 f217 f2170 f2171 f2172 f2173 f2174 f2175 f2176 f2177 f2178 f2179 f218 f2180 f2181 f2182 f2183 f2184 f2185 f2186 f2187 f2188 f2189 f219 f2190 f2191 f2192 f2193 f2194 f2195 f2196 f2197 f2198 f2199 f22 f220 f2200 f2201 f2202 f2203 f2204 f2205 f2206 f2207 f2208 f2209 f221 f2210 f2211 f2212 f2213 f2214 f2215 f2216 f2217 f2218 f2219 f222 f2220 f2221 f2222 f2223 f2224 f2225 f2226 f2227 f2228 f2229 f223 f2230 f2231 f2232 f2233 f2234 f2235 f2236 f2237 f2238 f2239 f224 f2240 f2241 f2242 f2243 f2244 f2245 f2246 f2247 f2248 f2249 f225 f2250 f2251 f2252 f2253 f2254 f2255 f2256 f2257 f2258 f2259 f226 f2260 f2261 f2262 f2263 f2264 f2265 f2266 f2267 f2268 f2269 f227 f2270 f2271 f2272 f2273 f2274 f2275 f2276 f2277 f2278 f2279 f228 f2280 f2281 f2282 f2283 f2284 f2285 f2286 f2287 f2288 f2289 f229 f2290 f2291 f2292 f2293 f2294 f2295 f2296 f2297 f2298 f2299 f23 f230 f2300 f2301 f2302 f2303 f2304 f2305 f2306 f2307 f2308 f2309 f231 f2310 f2311 f2312 f2313 f2314 f2315 f2316 f2317 f2318 f2319 f232 f2320 f2321 f2322 f2323 f2324 f2325 f2326 f2327 f2328 f2329 f233 f2330 f2331 f2332 f2333 f2334 f2335 f2336 f2337 f2338 f2339 f234 f2340 f2341 f2342 f2343 f2344 f2345 f2346 f2347 f2348 f2349 f235 f2350 f2351 f2352 f2353 f2354 f2355 f2356 f2357 f2358 f2359 f236 f2360 f2361 f2362 f2363 f2364 f2365 f2366 f2367 f2368 f2369 f237 f2370 f2371 f2372 f2373 f2374 f2375 f2376 f2377 f2378 f2379 f238 f2380 f2381 f2382 f2383 f2384 f2385 f2386 f2387 f2388 f2389 f239 f2390 f2391 f2392 f2393 f2394 f2395 f2396 f2397 f2398 f2399 f24 f240 f2400 f2401 f2402 f2403 f2404 f2405 f2406 f2407 f2408 f2409 f241 f2410 f2411 f2412 f2413 f2414 f2415 f2416 f2417 f2418 f2419 f242 f2420 f2421 f2422 f2423 f2424 f2425 f2426 f2427 f2428 f2429 f243 f2430 f2431 f2432 f2433 f2434 f2435 f2436 f2437 f2438 f2439 f244 f2440 f2441 f2442 f2443 f2444 f2445 f2446 f2447 f2448 f2449 f245 f2450 f2451 f2452 f2453 f2454 f2455 f2456 f2457 f2458 f2459 f246 f2460 f2461 f2462 f2463 f2464 f2465 f2466 f2467 f2468 f2469 f247 f2470 f2471 f2472 f2473 f2474 f2475 f2476 f2477 f2478 f2479 f248 f2480 f2481 f2482 f2483 f2484 f2485 f2486 f2487 f2488 f2489 f249 f2490 f2491 f2492 f2493 f2494 f2495 f2496 f2497 f2498 f2499 f25 f250 f2500 f2501 f2502 f2503 f2504 f2505 f2506 f2507 f2508 f2509 f251 f2510 f2511 f2512 f2513 f2514 f2515 f2516 f2517 f2518 f2519 f252 f2520 f2521 f2522 f2523 f2524 f2525 f2526 f2527 f2528 f2529 f253 f2530 f2531 f2532 f2533 f2534 f2535 f2536 f2537 f2538 f2539 f254 f2540 f2541 f2542 f2543 f2544 f2545 f2546 f2547 f2548 f2549 f255 f2550 f2551 f2552 f2553 f2554 f2555 f2556 f2557 f2558 f2559 f256 f2560 f2561 f2562 f2563 f2564 f2565 f2566 f2567 f2568 f2569 f257 f2570 f2571 f2572 f2573 f2574 f2575 f2576 f2577 f2578 f2579 f258 f2580 f2581 f2582 f2583 f2584 f2585 f2586 f2587 f2588 f2589 f259 f2590 f2591 f2592 f2593 f2594 f2595 f2596 f2597 f2598 f2599 f26 f260 f2600 f2601 f2602 f2603 f2604 f2605 f2606 f2607 f2608 f2609 f261 f2610 f2611 f2612 f2613 f2614 f2615 f2616 f2617 f2618 f2619 f262 f2620 f2621 f2622 f2623 f2624 f2625 f2626 f2627 f2628 f2629 f263 f2630 f2631 f2632 f2633 f2634 f2635 f2636 f2637 f2638 f2639 f264 f2640 f2641 f2642 f2643 f2644 f2645 f2646 f2647 f2648 f2649 f265 f2650 f2651 f2652 f2653 f2654 f2655 f2656 f2657 f2658 f2659 f266 f2660 f2661 f2662 f2663 f2664 f2665 f2666 f2667 f2668 f2669 f267 f2670 f2671 f2672 f2673 f2674 f2675 f2676 f2677 f2678 f2679 f268 f2680 f2681 f2682 f2683 f2684 f2685 f2686 f2687 f2688 f2689 f269 f2690 f2691 f2692 f2693 f2694 f2695 f2696 f2697 f2698 f2699 f27 f270 f2700 f2701 f2702 f2703 f2704 f2705 f2706 f2707 f2708 f2709 f271 f2710 f2711 f2712 f2713 f2714 f2715 f2716 f2717 f2718 f2719 f272 f2720 f2721 f2722 f2723 f2724 f2725 f2726 f2727 f2728 f2729 f273 f2730 f2731 f2732 f2733 f2734 f2735 f2736 f2737 f2738 f2739 f274 f2740 f2741 f2742 f2743 f2744 f2745 f2746 f2747 f2748 f2749 f275 f2750 f2751 f2752 f2753 f2754 f2755 f2756 f2757 f2758 f2759 f276 f2760 f2761 f2762 f2763 f2764 f2765 f2766 f2767 f2768 f2769 f277 f2770 f2771 f2772 f2773 f2774 f2775 f2776 f2777 f2778 f2779 f278 f2780 f2781 f2782 f2783 f2784 f2785 f2786 f2787 f2788 f2789 f279 f2790 f2791 f2792 f2793 f2794 f2795 f2796 f2797 f2798 f2799 f28 f280 f2800 f2801 f2802 f2803 f2804 f2805 f2806 f2807 f2808 f2809 f281 f2810 f2811 f2812 f2813 f2814 f2815 f2816 f2817 f2818 f2819 f282 f2820 f2821 f2822 f2823 f2824 f2825 f2826 f2827 f2828 f2829 f283 f2830 f2831 f2832 f2833 f2834 f2835 f2836 f2837 f2838 f2839 f284 f2840 f2841 f2842 f2843 f2844 f2845 f2846 f2847 f2848 f2849 f285 f2850 f2851 f2852 f2853 f2854 f2855 f2856 f2857 f2858 f2859 f286 f2860 f2861 f2862 f2863 f2864 f2865 f2866 f2867 f2868 f2869 f287 f2870 f2871 f2872 f2873 f2874 f2875 f2876 f2877 f2878 f2879 f288 f2880 f2881 f2882 f2883 f2884 f2885 f2886 f2887 f2888 f2889 f289 f2890 f2891 f2892 f2893 f2894 f2895 f2896 f2897 f2898 f2899 f29 f290 f2900 f2901 f2902 f2903 f2904 f2905 f2906 f2907 f2908 f2909 f291 f2910 f2911 f2912 f2913 f2914 f2915 f2916 f2917 f2918 f2919 f292 f2920 f2921 f2922 f2923 f2924 f2925 f2926 f2927 f2928 f2929 f293 f2930 f2931 f2932 f2933 f2934 f2935 f2936 f2937 f2938 f2939 f294 f2940 f2941 f2942 f2943 f2944 f2945 f2946 f2947 f2948 f2949 f295 f2950 f2951 f2952 f2953 f2954 f2955 f2956 f2957 f2958 f2959 f296 f2960 f2961 f2962 f2963 f2964 f2965 f2966 f2967 f2968 f2969 f297 f2970 f2971 f2972 f2973 f2974 f2975 f2976 f2977 f2978 f2979 f298 f2980 f2981 f2982 f2983 f2984 f2985 f2986 f2987 f2988 f2989 f299 f2990 f2991 f2992 f2993 f2994 f2995 f2996 f2997 f2998 f2999 f3 f30 f300 f301 f302 f303 f304 f305 f306 f307 f308 f309 f31 f310 f311 f312 f313 f314 f315 f316 f317 f318 f319 f32 f320 f321 f322 f323 f324 f325 f326 f327 f328 f329 f33 f330 f331 f332 f333 f334 f335 f336 f337 f338 f339 f34 f340 f341 f342 f343 f344 f345 f346 f347 f348 f349 f35 f350 f351 f352 f353 f354 f355 f356 f357 f358 f359 f36 f360 f361 f362 f363 f364 f365 f366 f367 f368 f369 f37 f370 f371 f372 f373 f374 f375 f376 f377 f378 f379 f38 f380 f381 f382 f383 f384 f385 f386 f387 f388 f389 f39 f390 f391 f392 f393 f394 f395 f396 f397 f398 f399 f4 f40 f400 f401 f402 f403 f404 f405 f406 f407 f408 f409 f41 f410 f411 f412 f413 f414 f415 f416 f417 f418 f419 f42 f420 f421 f422 f423 f424 f425 f426 f427 f428 f429 f43 f430 f431 f432 f433 f434 f435 f436 f437 f438 f439 f44 f440 f441 f442 f443 f444 f445 f446 f447 f448 f449 f45 f450 f451 f452 f453 f454 f455 f456 f457 f458 f459 f46 f460 f461 f462 f463 f464 f465 f466 f467 f468 f469 f47 f470 f471 f472 f473 f474 f475 f476 f477 f478 f479 f48 f480 f481 f482 f483 f484 f485 f486 f487 f488 f489 f49 f490 f491 f492 f493 f494 f495 f496 f497 f498 f499 f5 f50 f500 f501 f502 f503 f504 f505 f506 f507 f508 f509 f51 f510 f511 f512 f513 f514 f515 f516 f517 f518 f519 f52 f520 f521 f522 f523 f524 f525 f526 f527 f528 f529 f53 f530 f531 f532 f533 f534 f535 f536 f537 f538 f539 f54 f540 f541 f542 f543 f544 f545 f546 f547 f548 f549 f55 f550 f551 f552 f553 f554 f555 f556 f557 f558 f559 f56 f560 f561 f562 f563 f564 f565 f567 f568 f569 f57 f570 f571 f572 f573 f574 f575 f576 f577 f578 f579 f58 f580 f581 f582 f583 f584 f585 f586 f587 f588 f589 f59 f590 f591 f592 f593 f594 f595 f596 f597 f598 f599 f6 f60 f600 f601 f602 f603 f604 f605 f606 f607 f608 f609 f61 f610 f611 f612 f613 f614 f615 f616 f617 f618 f619 f62 f620 f621 f622 f623 f624 f625 f626 f627 f628 f629 f63 f630 f631 f632 f633 f634 f635 f636 f637 f638 f639 f64 f640 f641 f642 f643 f644 f645 f646 f647 f648 f649 f65 f650 f651 f652 f653 f654 f655 f656 f657 f658 f659 f66 f660 f661 f662 f663 f664 f665 f666 f667 f668 f669 f67 f670 f671 f672 f673 f674 f675 f676 f677 f678 f679 f68 f680 f681 f682 f683 f684 f685 f686 f687 f688 f689 f69 f690 f691 f692 f693 f694 f695 f696 f697 f698 f699 f7 f70 f700 f701 f702 f703 f704 f705 f706 f707 f708 f709 f71 f710 f711 f712 f713 f714 f715 f716 f717 f718 f719 f72 f720 f721 f722 f723 f724 f725 f726 f727 f728 f729 f73 f730 f731 f732 f733 f734 f735 f736 f737 f738 f739 f74 f740 f741 f742 f743 f744 f745 f746 f747 f748 f749 f75 f750 f751 f752 f753 f754 f755 f756 f757 f758 f759 f76 f760 f761 f762 f763 f764 f765 f766 f767 f768 f769 f77 f770 f771 f772 f773 f774 f775 f776 f777 f778 f779 f78 f780 f781 f782 f783 f784 f785 f786 f787 f788 f789 f79 f790 f791 f792 f793 f794 f795 f796 f797 f798 f799 f8 f80 f800 f801 f802 f803 f804 f805 f806 f807 f808 f809 f81 f810 f811 f812 f813 f814 f815 f816 f817 f818 f819 f82 f820 f821 f822 f823 f824 f825 f826 f827 f828 f829 f83 f830 f831 f832 f833 f834 f835 f836 f837 f838 f839 f84 f840 f841 f842 f843 f844 f845 f846 f847 f848 f849 f85 f850 f851 f852 f853 f854 f855 f856 f857 f858 f859 f86 f860 f861 f862 f863 f864 f865 f866 f867 f868 f869 f87 f870 f871 f872 f873 f874 f875 f876 f877 f878 f879 f88 f880 f881 f882 f883 f884 f885 f886 f887 f888 f889 f89 f890 f891 f892 f893 f894 f895 f896 f897 f898 f899 f9 f90 f900 f901 f902 f903 f904 f905 f906 f907 f908 f909 f91 f910 f911 f912 f913 f914 f915 f916 f917 f918 f919 f92 f920 f921 f922 f923 f924 f925 f926 f927 f928 f929 f93 f930 f931 f932 f933 f934 f935 f936 f937 f938 f939 f94 f940 f941 f942 f943 f944 f945 f946 f947 f948 f949 f95 f950 f951 f952 f953 f954 f955 f956 f957 f958 f959 f96 f960 f961 f962 f963 f964 f965 f966 f967 f968 f969 f97 f970 f971 f972 f973 f974 f975 f976 f977 f978 f979 f98 f980 f981 f982 f983 f984 f985 f986 f987 f988 f989 f99 f990 f991 f992 f993 f994 f995 f996 f997 f998 f999 ./d115.sanityn/f1523: ./d116.sanityn: tdir0 ./d116.sanityn/tdir0: tdir11 ./d116.sanityn/tdir0/tdir11: ./d117b.sanityn: ./d117c.sanityn: ./d117d.sanityn: tdir1 tdir2 ./d117d.sanityn/tdir1: ./d117d.sanityn/tdir2: ./d117e.sanityn: ./d117f.sanityn: ./d117g.sanityn: tfile4 ./d117h.sanityn: tdir2 tfile4 ./d117h.sanityn/tdir2: ./d117i.sanityn: d117i.sanityn f117i.sanityn ./d117i.sanityn/d117i.sanityn: ./d117j.sanityn: tdir1 tfile.1 tfile13 ./d117j.sanityn/tdir1: tdir11 tfile12 ./d117j.sanityn/tdir1/tdir11: ./d117k.sanityn: tdir1 tfile.1 ./d117k.sanityn/tdir1: tdir11 tfile12 tfile13 ./d117k.sanityn/tdir1/tdir11: ./d117l.sanityn: tdir2 tfile.1 ./d117l.sanityn/tdir2: tdir21 ./d117l.sanityn/tdir2/tdir21: ./d117m.sanityn: tdir2 tfile.1 ./d117m.sanityn/tdir2: tdir21 ./d117m.sanityn/tdir2/tdir21: ./d117n.sanityn: tdir2 ./d117n.sanityn/tdir2: tdir21 ./d117n.sanityn/tdir2/tdir21: ./d117o.sanityn: tdir1 ./d117o.sanityn/tdir1: ./d117pb.sanityn: ./d117p.sanityn: d117p.sanityn.0 d117p.sanityn.1 ./d117p.sanityn/d117p.sanityn.0: ./d117p.sanityn/d117p.sanityn.1: ./d117q.sanityn: striped_dir ./d117q.sanityn/striped_dir: ./d117r.sanityn: ./d117s.sanityn: ./d117t.sanityn: ./d117ub.sanityn: ./d117u.sanityn: ./d117w.sanityn: ./d117x.sanityn: tdir1 tdir2 tfile1 tfile2 ./d117x.sanityn/tdir1: ./d117x.sanityn/tdir2: ./d117ya.sanityn: lnkd5 lnkf4 tdir1 tdir2 tfile3 ./d117ya.sanityn/tdir1: tdir13 tfile11 tfile12 ./d117ya.sanityn/tdir1/tdir13: ./d117ya.sanityn/tdir2: ./d117yb.sanityn: etc ./d117yb.sanityn/etc: adjtime aliases aliases.db alternatives anacrontab asound.conf audit authselect bash_completion.d bashrc bindresvport.blacklist binfmt.d centos-release chkconfig.d cron.d cron.daily cron.deny cron.hourly cron.monthly crontab cron.weekly crypto-policies csh.cshrc csh.login dbus-1 debuginfod default depmod.d dhcp DIR_COLORS DIR_COLORS.256color DIR_COLORS.lightbgcolor dnf dracut.conf dracut.conf.d e2fsck.conf e2scrub.conf environment ethertypes exports exports.d filesystems firewalld fonts fstab gcrypt gdbinit gdbinit.d gnupg GREP_COLORS groff group group- grub.d gshadow gshadow- gss gssproxy host.conf hosts idmapd.conf init.d inittab inputrc iproute2 issue issue.net java jvm jvm-commmon kdump kdump.conf kdump.conf.rpmnew kernel keyutils krb5.conf krb5.conf.d ld.so.cache ld.so.conf ld.so.conf.d libaudit.conf libibverbs.d libnl libreport libssh libuser.conf localtime login.defs logrotate.conf logrotate.d machine-id magic mail.rc makedumpfile.conf.sample man_db.conf microcode_ctl mke2fs.conf modprobe.d modules-load.d motd motd.d mtab netconfig NetworkManager networks nfs.conf nfsmount.conf nftables nsswitch.conf nsswitch.conf.bak nsswitch.conf.rpmnew openldap opt os-release pam.d passwd passwd- pkcs11 pki pm polkit-1 popt.d postfix prelink.conf.d printcap profile profile.d protocols quilt.quiltrc rc0.d rc1.d rc2.d rc3.d rc4.d rc5.d rc6.d rc.d rc.local rdma redhat-release request-key.conf request-key.d resolv.conf rhsm rocky-release rocky-release-upstream rpc rpm rsyslog.conf rsyslog.conf.rpmnew rsyslog.d rwtab rwtab.d sasl2 security selinux services sestatus.conf shadow shadow- shells skel ssh ssl statetab statetab.d subgid subgid- sudo.conf sudoers sudoers.d sudo-ldap.conf swid sysconfig sysctl.conf sysctl.d systemd system-release system-release-cpe tcsd.conf terminfo tmpfiles.d tuned udev unbound virc wgetrc X11 xattr.conf xdg xinetd.d yum yum.conf yum.repos.d ./d117yb.sanityn/etc/alternatives: alt-java alt-java.1.gz easy_install-3 ifdown ifup java java.1.gz jjs jjs.1.gz jre jre_1.8.0 jre_1.8.0_openjdk jre_openjdk keytool keytool.1.gz ld libnssckbi.so.x86_64 mta mta-aliasesman mta-mailq mta-mailqman mta-newaliases mta-newaliasesman mta-pam mta-rmail mta-sendmail mta-sendmailman mta-smtpdman orbd orbd.1.gz pack200 pack200.1.gz pip-3 pip3 policytool policytool.1.gz pydoc-3 pydoc3 python python3 python3-config python3-man pyvenv-3 rmid rmid.1.gz rmiregistry rmiregistry.1.gz servertool servertool.1.gz tnameserv tnameserv.1.gz unpack200 unpack200.1.gz unversioned-python unversioned-python-man ./d117yb.sanityn/etc/audit: auditd.conf audit-stop.rules plugins.d rules.d ./d117yb.sanityn/etc/audit/plugins.d: af_unix.conf ./d117yb.sanityn/etc/audit/rules.d: audit.rules ./d117yb.sanityn/etc/authselect: custom user-nsswitch.conf ./d117yb.sanityn/etc/authselect/custom: ./d117yb.sanityn/etc/bash_completion.d: authselect-completion.sh iprconfig pip2 quilt zfs zpool ./d117yb.sanityn/etc/binfmt.d: ./d117yb.sanityn/etc/chkconfig.d: ./d117yb.sanityn/etc/cron.d: 0hourly ./d117yb.sanityn/etc/cron.daily: logrotate ./d117yb.sanityn/etc/cron.hourly: 0anacron ./d117yb.sanityn/etc/cron.monthly: ./d117yb.sanityn/etc/cron.weekly: ./d117yb.sanityn/etc/crypto-policies: back-ends config local.d policies state ./d117yb.sanityn/etc/crypto-policies/back-ends: bind.config gnutls.config java.config krb5.config libreswan.config libssh.config nss.config openssh.config opensshserver.config opensslcnf.config openssl.config ./d117yb.sanityn/etc/crypto-policies/local.d: nss-p11-kit.config ./d117yb.sanityn/etc/crypto-policies/policies: modules ./d117yb.sanityn/etc/crypto-policies/policies/modules: ./d117yb.sanityn/etc/crypto-policies/state: current CURRENT.pol ./d117yb.sanityn/etc/dbus-1: session.conf session.d system.conf system.d ./d117yb.sanityn/etc/dbus-1/session.d: ./d117yb.sanityn/etc/dbus-1/system.d: com.redhat.RHSM1.conf com.redhat.RHSM1.Facts.conf org.freedesktop.PolicyKit1.conf teamd.conf ./d117yb.sanityn/etc/debuginfod: elfutils.urls ./d117yb.sanityn/etc/default: useradd ./d117yb.sanityn/etc/depmod.d: dist.conf ./d117yb.sanityn/etc/dhcp: dhclient.conf dhclient.d ./d117yb.sanityn/etc/dhcp/dhclient.d: ./d117yb.sanityn/etc/dnf: aliases.d dnf.conf modules.d modules.defaults.d plugins protected.d vars ./d117yb.sanityn/etc/dnf/aliases.d: ./d117yb.sanityn/etc/dnf/modules.d: javapackages-runtime.module javapackages-tools.module libselinux-python.module perl-IO-Socket-SSL.module perl-libwww-perl.module perl.module python27.module python36.module python38.module swig.module ./d117yb.sanityn/etc/dnf/modules.defaults.d: ./d117yb.sanityn/etc/dnf/plugins: copr.conf copr.d debuginfo-install.conf product-id.conf subscription-manager.conf ./d117yb.sanityn/etc/dnf/plugins/copr.d: ./d117yb.sanityn/etc/dnf/protected.d: dnf.conf setup.conf sudo.conf systemd.conf yum.conf ./d117yb.sanityn/etc/dnf/vars: contentdir rltype sigcontentdir stream ./d117yb.sanityn/etc/dracut.conf.d: 50-network-legacy.conf ./d117yb.sanityn/etc/exports.d: ./d117yb.sanityn/etc/firewalld: firewalld.conf helpers icmptypes ipsets lockdown-whitelist.xml policies services zones ./d117yb.sanityn/etc/firewalld/helpers: ./d117yb.sanityn/etc/firewalld/icmptypes: ./d117yb.sanityn/etc/firewalld/ipsets: ./d117yb.sanityn/etc/firewalld/policies: ./d117yb.sanityn/etc/firewalld/services: ./d117yb.sanityn/etc/firewalld/zones: ./d117yb.sanityn/etc/fonts: conf.d fonts.conf ./d117yb.sanityn/etc/fonts/conf.d: 10-hinting-slight.conf 10-scale-bitmap-fonts.conf 20-unhint-small-dejavu-sans.conf 20-unhint-small-vera.conf 25-unhint-nonlatin.conf 30-metric-aliases.conf 40-nonlatin.conf 45-generic.conf 45-latin.conf 49-sansserif.conf 50-user.conf 51-local.conf 57-dejavu-sans.conf 60-generic.conf 60-latin.conf 65-fonts-persian.conf 65-nonlatin.conf 69-unifont.conf 80-delicious.conf 90-synthetic.conf README ./d117yb.sanityn/etc/gcrypt: random.conf ./d117yb.sanityn/etc/gdbinit.d: ./d117yb.sanityn/etc/gnupg: ./d117yb.sanityn/etc/groff: site-font site-tmac ./d117yb.sanityn/etc/groff/site-font: ./d117yb.sanityn/etc/groff/site-tmac: man.local mdoc.local ./d117yb.sanityn/etc/grub.d: 00_header 00_tuned 01_users 08_fallback_counting 10_linux 10_reset_boot_success 12_menu_auto_hide 20_linux_xen 20_ppc_terminfo 30_os-prober 30_uefi-firmware 40_custom 41_custom README ./d117yb.sanityn/etc/gss: mech.d ./d117yb.sanityn/etc/gss/mech.d: gssproxy.conf ./d117yb.sanityn/etc/gssproxy: 24-nfs-server.conf 99-nfs-client.conf gssproxy.conf ./d117yb.sanityn/etc/iproute2: bpf_pinning ematch_map group nl_protos rt_dsfield rt_protos rt_realms rt_scopes rt_tables ./d117yb.sanityn/etc/java: java-1.8.0-openjdk security ./d117yb.sanityn/etc/java/java-1.8.0-openjdk: java-1.8.0-openjdk-1.8.0.452.b09-2.el8.x86_64 ./d117yb.sanityn/etc/java/java-1.8.0-openjdk/java-1.8.0-openjdk-1.8.0.452.b09-2.el8.x86_64: lib ./d117yb.sanityn/etc/java/java-1.8.0-openjdk/java-1.8.0-openjdk-1.8.0.452.b09-2.el8.x86_64/lib: calendars.properties logging.properties security ./d117yb.sanityn/etc/java/java-1.8.0-openjdk/java-1.8.0-openjdk-1.8.0.452.b09-2.el8.x86_64/lib/security: blacklisted.certs cacerts java.policy java.security nss.cfg nss.fips.cfg policy ./d117yb.sanityn/etc/java/java-1.8.0-openjdk/java-1.8.0-openjdk-1.8.0.452.b09-2.el8.x86_64/lib/security/policy: limited unlimited ./d117yb.sanityn/etc/java/java-1.8.0-openjdk/java-1.8.0-openjdk-1.8.0.452.b09-2.el8.x86_64/lib/security/policy/limited: local_policy.jar US_export_policy.jar ./d117yb.sanityn/etc/java/java-1.8.0-openjdk/java-1.8.0-openjdk-1.8.0.452.b09-2.el8.x86_64/lib/security/policy/unlimited: local_policy.jar US_export_policy.jar ./d117yb.sanityn/etc/java/security: security.d ./d117yb.sanityn/etc/java/security/security.d: ./d117yb.sanityn/etc/jvm: ./d117yb.sanityn/etc/jvm-commmon: ./d117yb.sanityn/etc/kdump: post.d pre.d ./d117yb.sanityn/etc/kdump/post.d: ./d117yb.sanityn/etc/kdump/pre.d: ./d117yb.sanityn/etc/kernel: install.d postinst.d ./d117yb.sanityn/etc/kernel/install.d: 20-grubby.install 90-loaderentry.install ./d117yb.sanityn/etc/kernel/postinst.d: ./d117yb.sanityn/etc/keyutils: ./d117yb.sanityn/etc/krb5.conf.d: crypto-policies ./d117yb.sanityn/etc/ld.so.conf.d: bind-export-x86_64.conf kernel-4.18.0-513.18.1.el8_9.x86_64.conf kernel-4.18.0-553.34.1.el8_10.x86_64.conf kernel-4.18.0-553.58.1.el8_10.x86_64.conf ./d117yb.sanityn/etc/libibverbs.d: bnxt_re.driver cxgb4.driver efa.driver hfi1verbs.driver hns.driver irdma.driver mana.driver mlx4.driver mlx5.driver qedr.driver rxe.driver siw.driver vmw_pvrdma.driver ./d117yb.sanityn/etc/libnl: classid pktloc ./d117yb.sanityn/etc/libreport: events events.d plugins workflows.d ./d117yb.sanityn/etc/libreport/events: ./d117yb.sanityn/etc/libreport/events.d: collect_dnf.conf ./d117yb.sanityn/etc/libreport/plugins: ./d117yb.sanityn/etc/libreport/workflows.d: ./d117yb.sanityn/etc/libssh: libssh_client.config libssh_server.config ./d117yb.sanityn/etc/logrotate.d: btmp dnf firewalld kvm_stat subscription-manager syslog wtmp ./d117yb.sanityn/etc/microcode_ctl: ucode_with_caveats ./d117yb.sanityn/etc/microcode_ctl/ucode_with_caveats: ./d117yb.sanityn/etc/modprobe.d: firewalld-sysctls.conf lockd.conf mlx4.conf tuned.conf ./d117yb.sanityn/etc/modules-load.d: ./d117yb.sanityn/etc/motd.d: ./d117yb.sanityn/etc/NetworkManager: conf.d dispatcher.d dnsmasq.d dnsmasq-shared.d NetworkManager.conf system-connections ./d117yb.sanityn/etc/NetworkManager/conf.d: ./d117yb.sanityn/etc/NetworkManager/dispatcher.d: 11-dhclient no-wait.d pre-down.d pre-up.d ./d117yb.sanityn/etc/NetworkManager/dispatcher.d/no-wait.d: ./d117yb.sanityn/etc/NetworkManager/dispatcher.d/pre-down.d: ./d117yb.sanityn/etc/NetworkManager/dispatcher.d/pre-up.d: ./d117yb.sanityn/etc/NetworkManager/dnsmasq.d: ./d117yb.sanityn/etc/NetworkManager/dnsmasq-shared.d: ./d117yb.sanityn/etc/NetworkManager/system-connections: ./d117yb.sanityn/etc/nftables: main.nft nat.nft osf router.nft ./d117yb.sanityn/etc/nftables/osf: pf.os ./d117yb.sanityn/etc/openldap: certs ldap.conf ./d117yb.sanityn/etc/openldap/certs: ./d117yb.sanityn/etc/opt: ./d117yb.sanityn/etc/pam.d: config-util crond cvs fingerprint-auth login other passwd polkit-1 postlogin remote runuser runuser-l smartcard-auth smtp smtp.postfix sshd su subscription-manager sudo sudo-i su-l system-auth systemd-user vlock ./d117yb.sanityn/etc/pkcs11: modules ./d117yb.sanityn/etc/pkcs11/modules: ./d117yb.sanityn/etc/pki: ca-trust consumer entitlement java nssdb product product-default rpm-gpg rsyslog swid tls ./d117yb.sanityn/etc/pki/ca-trust: ca-legacy.conf extracted README source ./d117yb.sanityn/etc/pki/ca-trust/extracted: edk2 java openssl pem README ./d117yb.sanityn/etc/pki/ca-trust/extracted/edk2: cacerts.bin README ./d117yb.sanityn/etc/pki/ca-trust/extracted/java: cacerts README ./d117yb.sanityn/etc/pki/ca-trust/extracted/openssl: ca-bundle.trust.crt README ./d117yb.sanityn/etc/pki/ca-trust/extracted/pem: email-ca-bundle.pem objsign-ca-bundle.pem README tls-ca-bundle.pem ./d117yb.sanityn/etc/pki/ca-trust/source: anchors blacklist ca-bundle.legacy.crt README ./d117yb.sanityn/etc/pki/ca-trust/source/anchors: ./d117yb.sanityn/etc/pki/ca-trust/source/blacklist: ./d117yb.sanityn/etc/pki/consumer: cert.pem key.pem ./d117yb.sanityn/etc/pki/entitlement: 3434645747755415203-key.pem 3434645747755415203.pem ./d117yb.sanityn/etc/pki/java: cacerts ./d117yb.sanityn/etc/pki/nssdb: cert8.db cert9.db key3.db key4.db pkcs11.txt secmod.db ./d117yb.sanityn/etc/pki/product: 479.pem 491.pem ./d117yb.sanityn/etc/pki/product-default: ./d117yb.sanityn/etc/pki/rpm-gpg: RPM-GPG-KEY-EPEL-8 RPM-GPG-KEY-Rocky-8 RPM-GPG-KEY-Rocky-8-Testing RPM-GPG-KEY-rockyofficial RPM-GPG-KEY-rockytesting ./d117yb.sanityn/etc/pki/rsyslog: ./d117yb.sanityn/etc/pki/swid: CA ./d117yb.sanityn/etc/pki/swid/CA: ./d117yb.sanityn/etc/pki/tls: cert.pem certs ct_log_list.cnf misc openssl.cnf private ./d117yb.sanityn/etc/pki/tls/certs: ca-bundle.crt ca-bundle.trust.crt postfix.pem ./d117yb.sanityn/etc/pki/tls/misc: ./d117yb.sanityn/etc/pki/tls/private: postfix.key ./d117yb.sanityn/etc/pm: config.d power.d sleep.d ./d117yb.sanityn/etc/pm/config.d: ./d117yb.sanityn/etc/pm/power.d: ./d117yb.sanityn/etc/pm/sleep.d: ./d117yb.sanityn/etc/polkit-1: localauthority localauthority.conf.d rules.d ./d117yb.sanityn/etc/polkit-1/localauthority: 10-vendor.d 20-org.d 30-site.d 50-local.d 90-mandatory.d ./d117yb.sanityn/etc/polkit-1/localauthority/10-vendor.d: ./d117yb.sanityn/etc/polkit-1/localauthority/20-org.d: ./d117yb.sanityn/etc/polkit-1/localauthority/30-site.d: ./d117yb.sanityn/etc/polkit-1/localauthority/50-local.d: ./d117yb.sanityn/etc/polkit-1/localauthority/90-mandatory.d: ./d117yb.sanityn/etc/polkit-1/localauthority.conf.d: ./d117yb.sanityn/etc/polkit-1/rules.d: 49-polkit-pkla-compat.rules 50-default.rules ./d117yb.sanityn/etc/popt.d: ./d117yb.sanityn/etc/postfix: access canonical dynamicmaps.cf dynamicmaps.cf.d generic header_checks main.cf main.cf.proto master.cf.proto postfix-files postfix-files.d relocated transport virtual ./d117yb.sanityn/etc/postfix/dynamicmaps.cf.d: ./d117yb.sanityn/etc/postfix/postfix-files.d: ./d117yb.sanityn/etc/prelink.conf.d: ./d117yb.sanityn/etc/profile.d: colorgrep.csh colorgrep.sh colorls.csh colorls.sh colorxzgrep.csh colorxzgrep.sh colorzgrep.csh colorzgrep.sh csh.local cvs.csh cvs.sh debuginfod.csh debuginfod.sh gawk.csh gawk.sh lang.csh lang.sh less.csh less.sh sh.local which2.csh which2.sh ./d117yb.sanityn/etc/rc.d: init.d rc0.d rc1.d rc2.d rc3.d rc4.d rc5.d rc6.d rc.local ./d117yb.sanityn/etc/rc.d/init.d: functions README ./d117yb.sanityn/etc/rc.d/rc0.d: ./d117yb.sanityn/etc/rc.d/rc1.d: ./d117yb.sanityn/etc/rc.d/rc2.d: ./d117yb.sanityn/etc/rc.d/rc3.d: ./d117yb.sanityn/etc/rc.d/rc4.d: ./d117yb.sanityn/etc/rc.d/rc5.d: ./d117yb.sanityn/etc/rc.d/rc6.d: ./d117yb.sanityn/etc/rdma: mlx4.conf modules ./d117yb.sanityn/etc/rdma/modules: infiniband.conf iwarp.conf opa.conf rdma.conf roce.conf ./d117yb.sanityn/etc/request-key.d: id_resolver.conf ./d117yb.sanityn/etc/rhsm: ca facts logging.conf pluginconf.d rhsm.conf rhsm.conf.rpmnew syspurpose ./d117yb.sanityn/etc/rhsm/ca: redhat-entitlement-authority.pem redhat-uep.pem ./d117yb.sanityn/etc/rhsm/facts: ./d117yb.sanityn/etc/rhsm/pluginconf.d: ./d117yb.sanityn/etc/rhsm/syspurpose: syspurpose.json valid_fields.json ./d117yb.sanityn/etc/rpm: ./d117yb.sanityn/etc/rsyslog.d: ./d117yb.sanityn/etc/rwtab.d: gssproxy logrotate lustretests rsyslog ssh ./d117yb.sanityn/etc/sasl2: smtpd.conf ./d117yb.sanityn/etc/security: access.conf chroot.conf console.apps console.handlers console.perms console.perms.d faillock.conf group.conf limits.conf limits.d namespace.conf namespace.d namespace.init opasswd pam_env.conf pwhistory.conf pwquality.conf pwquality.conf.d sepermit.conf time.conf ./d117yb.sanityn/etc/security/console.apps: config-util subscription-manager ./d117yb.sanityn/etc/security/console.perms.d: ./d117yb.sanityn/etc/security/limits.d: ./d117yb.sanityn/etc/security/namespace.d: ./d117yb.sanityn/etc/security/pwquality.conf.d: ./d117yb.sanityn/etc/selinux: config semanage.conf targeted ./d117yb.sanityn/etc/selinux/targeted: booleans.subs_dist contexts logins policy setrans.conf seusers ./d117yb.sanityn/etc/selinux/targeted/contexts: customizable_types dbus_contexts default_contexts default_type failsafe_context files initrc_context lxc_contexts openssh_contexts removable_context securetty_types sepgsql_contexts snapperd_contexts systemd_contexts userhelper_context users virtual_domain_context virtual_image_context x_contexts ./d117yb.sanityn/etc/selinux/targeted/contexts/files: file_contexts file_contexts.homedirs file_contexts.local file_contexts.pre file_contexts.subs file_contexts.subs_dist media ./d117yb.sanityn/etc/selinux/targeted/contexts/users: guest_u root staff_u sysadm_u unconfined_u user_u xguest_u ./d117yb.sanityn/etc/selinux/targeted/logins: ./d117yb.sanityn/etc/selinux/targeted/policy: policy.31 ./d117yb.sanityn/etc/skel: ./d117yb.sanityn/etc/ssh: moduli ssh_config ssh_config.d ssh_config.rpmnew sshd_config ssh_host_dsa_key ssh_host_dsa_key.pub ssh_host_ecdsa_key ssh_host_ecdsa_key.pub ssh_host_ed25519_key ssh_host_ed25519_key.pub ssh_host_rsa_key ssh_host_rsa_key.pub ./d117yb.sanityn/etc/ssh/ssh_config.d: 05-redhat.conf ./d117yb.sanityn/etc/ssl: certs ./d117yb.sanityn/etc/statetab.d: kdump ./d117yb.sanityn/etc/sudoers.d: ./d117yb.sanityn/etc/swid: ./d117yb.sanityn/etc/sysconfig: console cpupower crond ebtables-config firewalld grub ip6tables-config iptables-config irqbalance kdump kdump.rpmnew man-db modules network-scripts nftables.conf readonly-root rngd rpcbind rsyslog run-parts selinux sshd zfs ./d117yb.sanityn/etc/sysconfig/console: ./d117yb.sanityn/etc/sysconfig/modules: ./d117yb.sanityn/etc/sysconfig/network-scripts: ifcfg-ens2 ./d117yb.sanityn/etc/sysctl.d: 99-sysctl.conf ./d117yb.sanityn/etc/systemd: coredump.conf journald.conf logind.conf pstore.conf resolved.conf system system.conf user user.conf ./d117yb.sanityn/etc/systemd/system: basic.target.wants dbus-org.freedesktop.NetworkManager.service dbus-org.freedesktop.nm-dispatcher.service getty.target.wants local-fs.target.wants multi-user.target.wants network-online.target.wants nfs-blkmap.service.requires nfs-idmapd.service.requires nfs-mountd.service.requires nfs-server.service.requires proc-sys-fs-binfmt_misc.mount remote-fs.target.wants rpc-gssd.service.requires rpc-statd-notify.service.requires rpc-statd.service.requires sockets.target.wants sysinit.target.wants syslog.service timers.target.wants ./d117yb.sanityn/etc/systemd/system/basic.target.wants: microcode.service ./d117yb.sanityn/etc/systemd/system/getty.target.wants: getty@tty1.service serial-getty@ttyS0.service ./d117yb.sanityn/etc/systemd/system/local-fs.target.wants: readonly-root.service ./d117yb.sanityn/etc/systemd/system/multi-user.target.wants: auditd.service crond.service dnf-makecache.timer irqbalance.service kdump.service NetworkManager.service nfs-client.target remote-fs.target rpcbind.service rsyslog.service sshd.service tuned.service ./d117yb.sanityn/etc/systemd/system/network-online.target.wants: NetworkManager-wait-online.service ./d117yb.sanityn/etc/systemd/system/nfs-blkmap.service.requires: nfs-convert.service ./d117yb.sanityn/etc/systemd/system/nfs-idmapd.service.requires: nfs-convert.service ./d117yb.sanityn/etc/systemd/system/nfs-mountd.service.requires: nfs-convert.service ./d117yb.sanityn/etc/systemd/system/nfs-server.service.requires: nfs-convert.service ./d117yb.sanityn/etc/systemd/system/remote-fs.target.wants: nfs-client.target ./d117yb.sanityn/etc/systemd/system/rpc-gssd.service.requires: nfs-convert.service ./d117yb.sanityn/etc/systemd/system/rpc-statd-notify.service.requires: nfs-convert.service ./d117yb.sanityn/etc/systemd/system/rpc-statd.service.requires: nfs-convert.service ./d117yb.sanityn/etc/systemd/system/sockets.target.wants: rpcbind.socket ./d117yb.sanityn/etc/systemd/system/sysinit.target.wants: import-state.service loadmodules.service nis-domainname.service rngd.service selinux-autorelabel-mark.service ./d117yb.sanityn/etc/systemd/system/timers.target.wants: unbound-anchor.timer ./d117yb.sanityn/etc/systemd/user: ./d117yb.sanityn/etc/terminfo: ./d117yb.sanityn/etc/tmpfiles.d: ./d117yb.sanityn/etc/tuned: active_profile bootcmdline post_loaded_profile profile_mode recommend.d tuned-main.conf ./d117yb.sanityn/etc/tuned/recommend.d: ./d117yb.sanityn/etc/udev: hwdb.bin hwdb.d rules.d udev.conf ./d117yb.sanityn/etc/udev/hwdb.d: ./d117yb.sanityn/etc/udev/rules.d: 99-lustre-test.rules ./d117yb.sanityn/etc/unbound: icannbundle.pem root.key ./d117yb.sanityn/etc/X11: applnk fontpath.d xinit xorg.conf.d ./d117yb.sanityn/etc/X11/applnk: ./d117yb.sanityn/etc/X11/fontpath.d: xorg-x11-fonts-Type1 ./d117yb.sanityn/etc/X11/xinit: xinitrc.d ./d117yb.sanityn/etc/X11/xinit/xinitrc.d: 50-systemd-user.sh ./d117yb.sanityn/etc/X11/xorg.conf.d: ./d117yb.sanityn/etc/xdg: autostart systemd ./d117yb.sanityn/etc/xdg/autostart: ./d117yb.sanityn/etc/xdg/systemd: user ./d117yb.sanityn/etc/xinetd.d: ./d117yb.sanityn/etc/yum: pluginconf.d protected.d vars ./d117yb.sanityn/etc/yum.repos.d: epel-modular.repo epel.repo epel-testing-modular.repo epel-testing.repo redhat.repo Rocky-AppStream.repo Rocky-BaseOS.repo Rocky-Debuginfo.repo Rocky-Devel.repo Rocky-Extras.repo Rocky-HighAvailability.repo Rocky-Media.repo Rocky-NFV.repo Rocky-Plus.repo Rocky-PowerTools.repo Rocky-ResilientStorage.repo Rocky-RT.repo Rocky-Sources.repo ./d117yc.sanityn: lnkd4 lnkf3 tdir1 tfile1 ./d117yc.sanityn/tdir1: ./d117za.sanityn: ./d117zb.sanityn: ./d117zc.sanityn: striped_dir ./d117zc.sanityn/striped_dir: ./d117zd.sanityn: striped_dir ./d117zd.sanityn/striped_dir: ./d118Aa.sanityn: ./d118Ab.sanityn: ./d118Ba.sanityn: ./d118Bb.sanityn: ./d118C.sanityn: f118C.sanityn ./d118d.sanityn: ./d118f.sanityn: ./d118g.sanityn: ./d118j.sanityn: ./d118k.sanityn: ./d118l.sanityn: ./d118m.sanityn: ./d118o.sanityn: f118o.sanityn ./d13: ./d14aa.sanityn: f14aa.sanityn ./d14ab.sanityn: sleep ./d14b.sanityn: sleep ./d14c.sanityn: sleep ./d14d.sanityn: sleep ./d16h.sanityn: f16h.sanityn ./d16i.sanityn: f16i.sanityn ./d20.sanityn: f20.sanityn ./d30.sanityn: bash ./d31a.sanityn: f31a.sanityn ./d31b.sanityn: f31b.sanityn ./d31s.sanityn: d31s.sanityn.old ./d31s.sanityn/d31s.sanityn.old: f31s.sanityn ./d33c.sanityn: local.3 remote remote.3 striped ./d33c.sanityn/local.3: ./d33c.sanityn/remote: ./d33c.sanityn/remote.3: ./d33c.sanityn/striped: subfile ./d35.sanityn: a0 a1 a10 a100 a1000 a1001 a1002 a1003 a1004 a1005 a1006 a1007 a1008 a1009 a101 a1010 a1011 a1012 a1013 a1014 a1015 a1016 a1017 a1018 a1019 a102 a1020 a1021 a1022 a1023 a1024 a1025 a1026 a1027 a1028 a1029 a103 a1030 a1031 a1032 a1033 a1034 a1035 a1036 a1037 a1038 a1039 a104 a1040 a1041 a1042 a1043 a1044 a1045 a1046 a1047 a1048 a1049 a105 a1050 a1051 a1052 a1053 a1054 a1055 a1056 a1057 a1058 a1059 a106 a1060 a1061 a1062 a1063 a1064 a1065 a1066 a1067 a1068 a1069 a107 a1070 a1071 a1072 a1073 a1074 a1075 a1076 a1077 a1078 a1079 a108 a1080 a1081 a1082 a1083 a1084 a1085 a1086 a1087 a1088 a1089 a109 a1090 a1091 a1092 a1093 a1094 a1095 a1096 a1097 a1098 a1099 a11 a110 a1100 a1101 a1102 a1103 a1104 a1105 a1106 a1107 a1108 a1109 a111 a1110 a1111 a1112 a1113 a1114 a1115 a1116 a1117 a1118 a1119 a112 a1120 a1121 a1122 a1123 a1124 a1125 a1126 a1127 a1128 a1129 a113 a1130 a1131 a1132 a1133 a1134 a1135 a1136 a1137 a1138 a1139 a114 a1140 a1141 a1142 a1143 a1144 a1145 a1146 a1147 a1148 a1149 a115 a1150 a1151 a1152 a1153 a1154 a1155 a1156 a1157 a1158 a1159 a116 a1160 a1161 a1162 a1163 a1164 a1165 a1166 a1167 a1168 a1169 a117 a1170 a1171 a1172 a1173 a1174 a1175 a1176 a1177 a1178 a1179 a118 a1180 a1181 a1182 a1183 a1184 a1185 a1186 a1187 a1188 a1189 a119 a1190 a1191 a1192 a1193 a1194 a1195 a1196 a1197 a1198 a1199 a12 a120 a1200 a1201 a1202 a1203 a1204 a1205 a1206 a1207 a1208 a1209 a121 a1210 a1211 a1212 a1213 a1214 a1215 a1216 a1217 a1218 a1219 a122 a1220 a1221 a1222 a1223 a1224 a1225 a1226 a1227 a1228 a1229 a123 a1230 a1231 a1232 a1233 a1234 a1235 a1236 a1237 a1238 a1239 a124 a1240 a1241 a1242 a1243 a1244 a1245 a1246 a1247 a1248 a1249 a125 a1250 a1251 a1252 a1253 a1254 a1255 a1256 a1257 a1258 a1259 a126 a1260 a1261 a1262 a1263 a1264 a1265 a1266 a1267 a1268 a1269 a127 a1270 a1271 a1272 a1273 a1274 a1275 a1276 a1277 a1278 a1279 a128 a1280 a1281 a1282 a1283 a1284 a1285 a1286 a1287 a1288 a1289 a129 a1290 a1291 a1292 a1293 a1294 a1295 a1296 a1297 a1298 a1299 a13 a130 a1300 a1301 a1302 a1303 a1304 a1305 a1306 a1307 a1308 a1309 a131 a1310 a1311 a1312 a1313 a1314 a1315 a1316 a1317 a1318 a1319 a132 a1320 a1321 a1322 a1323 a1324 a1325 a1326 a1327 a1328 a1329 a133 a1330 a1331 a1332 a1333 a1334 a1335 a1336 a1337 a1338 a1339 a134 a1340 a1341 a1342 a1343 a1344 a1345 a1346 a1347 a1348 a1349 a135 a1350 a1351 a1352 a1353 a1354 a1355 a1356 a1357 a1358 a1359 a136 a1360 a1361 a1362 a1363 a1364 a1365 a1366 a1367 a1368 a1369 a137 a1370 a1371 a1372 a1373 a1374 a1375 a1376 a1377 a1378 a1379 a138 a1380 a1381 a1382 a1383 a1384 a1385 a1386 a1387 a1388 a1389 a139 a1390 a1391 a1392 a1393 a1394 a1395 a1396 a1397 a1398 a1399 a14 a140 a1400 a1401 a1402 a1403 a1404 a1405 a1406 a1407 a1408 a1409 a141 a1410 a1411 a1412 a1413 a1414 a1415 a1416 a1417 a1418 a1419 a142 a1420 a1421 a1422 a1423 a1424 a1425 a1426 a1427 a1428 a1429 a143 a1430 a1431 a1432 a1433 a1434 a1435 a1436 a1437 a1438 a1439 a144 a1440 a1441 a1442 a1443 a1444 a1445 a1446 a1447 a1448 a1449 a145 a1450 a1451 a1452 a1453 a1454 a1455 a1456 a1457 a1458 a1459 a146 a1460 a1461 a1462 a1463 a1464 a1465 a1466 a1467 a1468 a1469 a147 a1470 a1471 a1472 a1473 a1474 a1475 a1476 a1477 a1478 a1479 a148 a1480 a1481 a1482 a1483 a1484 a1485 a1486 a1487 a1488 a1489 a149 a1490 a1491 a1492 a1493 a1494 a1495 a1496 a1497 a1498 a1499 a15 a150 a1500 a1501 a1502 a1503 a1504 a1505 a1506 a1507 a1508 a1509 a151 a1510 a1511 a1512 a1513 a1514 a1515 a1516 a1517 a1518 a1519 a152 a1520 a1521 a1522 a1523 a1524 a1525 a1526 a1527 a1528 a1529 a153 a1530 a1531 a1532 a1533 a1534 a1535 a1536 a1537 a1538 a1539 a154 a1540 a1541 a1542 a1543 a1544 a1545 a1546 a1547 a1548 a1549 a155 a1550 a1551 a1552 a1553 a1554 a1555 a1556 a1557 a1558 a1559 a156 a1560 a1561 a1562 a1563 a1564 a1565 a1566 a1567 a1568 a1569 a157 a1570 a1571 a1572 a1573 a1574 a1575 a1576 a1577 a1578 a1579 a158 a1580 a1581 a1582 a1583 a1584 a1585 a1586 a1587 a1588 a1589 a159 a1590 a1591 a1592 a1593 a1594 a1595 a1596 a1597 a1598 a1599 a16 a160 a1600 a1601 a1602 a1603 a1604 a1605 a1606 a1607 a1608 a1609 a161 a1610 a1611 a1612 a1613 a1614 a1615 a1616 a1617 a1618 a1619 a162 a1620 a1621 a1622 a1623 a1624 a1625 a1626 a1627 a1628 a1629 a163 a1630 a1631 a1632 a1633 a1634 a1635 a1636 a1637 a1638 a1639 a164 a1640 a1641 a1642 a1643 a1644 a1645 a1646 a1647 a1648 a1649 a165 a1650 a1651 a1652 a1653 a1654 a1655 a1656 a1657 a1658 a1659 a166 a1660 a1661 a1662 a1663 a1664 a1665 a1666 a1667 a1668 a1669 a167 a1670 a1671 a1672 a1673 a1674 a1675 a1676 a1677 a1678 a1679 a168 a1680 a1681 a1682 a1683 a1684 a1685 a1686 a1687 a1688 a1689 a169 a1690 a1691 a1692 a1693 a1694 a1695 a1696 a1697 a1698 a1699 a17 a170 a1700 a1701 a1702 a1703 a1704 a1705 a1706 a1707 a1708 a1709 a171 a1710 a1711 a1712 a1713 a1714 a1715 a1716 a1717 a1718 a1719 a172 a1720 a1721 a1722 a1723 a1724 a1725 a1726 a1727 a1728 a1729 a173 a1730 a1731 a1732 a1733 a1734 a1735 a1736 a1737 a1738 a1739 a174 a1740 a1741 a1742 a1743 a1744 a1745 a1746 a1747 a1748 a1749 a175 a1750 a1751 a1752 a1753 a1754 a1755 a1756 a1757 a1758 a1759 a176 a1760 a1761 a1762 a1763 a1764 a1765 a1766 a1767 a1768 a1769 a177 a1770 a1771 a1772 a1773 a1774 a1775 a1776 a1777 a1778 a1779 a178 a1780 a1781 a1782 a1783 a1784 a1785 a1786 a1787 a1788 a1789 a179 a1790 a1791 a1792 a1793 a1794 a1795 a1796 a1797 a1798 a1799 a18 a180 a1800 a1801 a1802 a1803 a1804 a1805 a1806 a1807 a1808 a1809 a181 a1810 a1811 a1812 a1813 a1814 a1815 a1816 a1817 a1818 a1819 a182 a1820 a1821 a1822 a1823 a1824 a1825 a1826 a1827 a1828 a1829 a183 a1830 a1831 a1832 a1833 a1834 a1835 a1836 a1837 a1838 a1839 a184 a1840 a1841 a1842 a1843 a1844 a1845 a1846 a1847 a1848 a1849 a185 a1850 a1851 a1852 a1853 a1854 a1855 a1856 a1857 a1858 a1859 a186 a1860 a1861 a1862 a1863 a1864 a1865 a1866 a1867 a1868 a1869 a187 a1870 a1871 a1872 a1873 a1874 a1875 a1876 a1877 a1878 a1879 a188 a1880 a1881 a1882 a1883 a1884 a1885 a1886 a1887 a1888 a1889 a189 a1890 a1891 a1892 a1893 a1894 a1895 a1896 a1897 a1898 a1899 a19 a190 a1900 a1901 a1902 a1903 a1904 a1905 a1906 a1907 a1908 a1909 a191 a1910 a1911 a1912 a1913 a1914 a1915 a1916 a1917 a1918 a1919 a192 a1920 a1921 a1922 a1923 a1924 a1925 a1926 a1927 a1928 a1929 a193 a1930 a1931 a1932 a1933 a1934 a1935 a1936 a1937 a1938 a1939 a194 a1940 a1941 a1942 a1943 a1944 a1945 a1946 a1947 a1948 a1949 a195 a1950 a1951 a1952 a1953 a1954 a1955 a1956 a1957 a1958 a1959 a196 a1960 a1961 a1962 a1963 a1964 a1965 a1966 a1967 a1968 a1969 a197 a1970 a1971 a1972 a1973 a1974 a1975 a1976 a1977 a1978 a1979 a198 a1980 a1981 a1982 a1983 a1984 a1985 a1986 a1987 a1988 a1989 a199 a1990 a1991 a1992 a1993 a1994 a1995 a1996 a1997 a1998 a1999 a2 a20 a200 a2000 a2001 a2002 a2003 a2004 a2005 a2006 a2007 a2008 a2009 a201 a2010 a2011 a2012 a2013 a2014 a2015 a2016 a2017 a2018 a2019 a202 a2020 a2021 a2022 a2023 a2024 a2025 a2026 a2027 a2028 a2029 a203 a2030 a2031 a2032 a2033 a2034 a2035 a2036 a2037 a2038 a2039 a204 a2040 a2041 a2042 a2043 a2044 a2045 a2046 a2047 a2048 a2049 a205 a2050 a2051 a2052 a2053 a2054 a2055 a2056 a2057 a2058 a2059 a206 a2060 a2061 a2062 a2063 a2064 a2065 a2066 a2067 a2068 a2069 a207 a2070 a2071 a2072 a2073 a2074 a2075 a2076 a2077 a2078 a2079 a208 a2080 a2081 a2082 a2083 a2084 a2085 a2086 a2087 a2088 a2089 a209 a2090 a2091 a2092 a2093 a2094 a2095 a2096 a2097 a2098 a2099 a21 a210 a2100 a2101 a2102 a2103 a2104 a2105 a2106 a2107 a2108 a2109 a211 a2110 a2111 a2112 a2113 a2114 a2115 a2116 a2117 a2118 a2119 a212 a2120 a2121 a2122 a2123 a2124 a2125 a2126 a2127 a2128 a2129 a213 a2130 a2131 a2132 a2133 a2134 a2135 a2136 a2137 a2138 a2139 a214 a2140 a2141 a2142 a2143 a2144 a2145 a2146 a2147 a2148 a2149 a215 a2150 a2151 a2152 a2153 a2154 a2155 a2156 a2157 a2158 a2159 a216 a2160 a2161 a2162 a2163 a2164 a2165 a2166 a2167 a2168 a2169 a217 a2170 a2171 a2172 a2173 a2174 a2175 a2176 a2177 a2178 a2179 a218 a2180 a2181 a2182 a2183 a2184 a2185 a2186 a2187 a2188 a2189 a219 a2190 a2191 a2192 a2193 a2194 a2195 a2196 a2197 a2198 a2199 a22 a220 a2200 a2201 a2202 a2203 a2204 a2205 a2206 a2207 a2208 a2209 a221 a2210 a2211 a2212 a2213 a2214 a2215 a2216 a2217 a2218 a2219 a222 a2220 a2221 a2222 a2223 a2224 a2225 a2226 a2227 a2228 a2229 a223 a2230 a2231 a2232 a2233 a2234 a2235 a2236 a2237 a2238 a2239 a224 a2240 a2241 a2242 a2243 a2244 a2245 a2246 a2247 a2248 a2249 a225 a2250 a2251 a2252 a2253 a2254 a2255 a2256 a2257 a2258 a2259 a226 a2260 a2261 a2262 a2263 a2264 a2265 a2266 a2267 a2268 a2269 a227 a2270 a2271 a2272 a2273 a2274 a2275 a2276 a2277 a2278 a2279 a228 a2280 a2281 a2282 a2283 a2284 a2285 a2286 a2287 a2288 a2289 a229 a2290 a2291 a2292 a2293 a2294 a2295 a2296 a2297 a2298 a2299 a23 a230 a2300 a2301 a2302 a2303 a2304 a2305 a2306 a2307 a2308 a2309 a231 a2310 a2311 a2312 a2313 a2314 a2315 a2316 a2317 a2318 a2319 a232 a2320 a2321 a2322 a2323 a2324 a2325 a2326 a2327 a2328 a2329 a233 a2330 a2331 a2332 a2333 a2334 a2335 a2336 a2337 a2338 a2339 a234 a2340 a2341 a2342 a2343 a2344 a2345 a2346 a2347 a2348 a2349 a235 a2350 a2351 a2352 a2353 a2354 a2355 a2356 a2357 a2358 a2359 a236 a2360 a2361 a2362 a2363 a2364 a2365 a2366 a2367 a2368 a2369 a237 a2370 a2371 a2372 a2373 a2374 a2375 a2376 a2377 a2378 a2379 a238 a2380 a2381 a2382 a2383 a2384 a2385 a2386 a2387 a2388 a2389 a239 a2390 a2391 a2392 a2393 a2394 a2395 a2396 a2397 a2398 a2399 a24 a240 a2400 a2401 a2402 a2403 a2404 a2405 a2406 a2407 a2408 a2409 a241 a2410 a2411 a2412 a2413 a2414 a2415 a2416 a2417 a2418 a2419 a242 a2420 a2421 a2422 a2423 a2424 a2425 a2426 a2427 a2428 a2429 a243 a2430 a2431 a2432 a2433 a2434 a2435 a2436 a2437 a2438 a2439 a244 a2440 a2441 a2442 a2443 a2444 a2445 a2446 a2447 a2448 a2449 a245 a2450 a2451 a2452 a2453 a2454 a2455 a2456 a2457 a2458 a2459 a246 a2460 a2461 a2462 a2463 a2464 a2465 a2466 a2467 a2468 a2469 a247 a2470 a2471 a2472 a2473 a2474 a2475 a2476 a2477 a2478 a2479 a248 a2480 a2481 a2482 a2483 a2484 a2485 a2486 a2487 a2488 a2489 a249 a2490 a2491 a2492 a2493 a2494 a2495 a2496 a2497 a2498 a2499 a25 a250 a2500 a2501 a2502 a2503 a2504 a2505 a2506 a2507 a2508 a2509 a251 a2510 a2511 a2512 a2513 a2514 a2515 a2516 a2517 a2518 a2519 a252 a2520 a2521 a2522 a2523 a2524 a2525 a2526 a2527 a2528 a2529 a253 a2530 a2531 a2532 a2533 a2534 a2535 a2536 a2537 a2538 a2539 a254 a2540 a2541 a2542 a2543 a2544 a2545 a2546 a2547 a2548 a2549 a255 a2550 a2551 a2552 a2553 a2554 a2555 a2556 a2557 a2558 a2559 a256 a2560 a2561 a2562 a2563 a2564 a2565 a2566 a2567 a2568 a2569 a257 a2570 a2571 a2572 a2573 a2574 a2575 a2576 a2577 a2578 a2579 a258 a2580 a2581 a2582 a2583 a2584 a2585 a2586 a2587 a2588 a2589 a259 a2590 a2591 a2592 a2593 a2594 a2595 a2596 a2597 a2598 a2599 a26 a260 a2600 a2601 a2602 a2603 a2604 a2605 a2606 a2607 a2608 a2609 a261 a2610 a2611 a2612 a2613 a2614 a2615 a2616 a2617 a2618 a2619 a262 a2620 a2621 a2622 a2623 a2624 a2625 a2626 a2627 a2628 a2629 a263 a2630 a2631 a2632 a2633 a2634 a2635 a2636 a2637 a2638 a2639 a264 a2640 a2641 a2642 a2643 a2644 a2645 a2646 a2647 a2648 a2649 a265 a2650 a2651 a2652 a2653 a2654 a2655 a2656 a2657 a2658 a2659 a266 a2660 a2661 a2662 a2663 a2664 a2665 a2666 a2667 a2668 a2669 a267 a2670 a2671 a2672 a2673 a2674 a2675 a2676 a2677 a2678 a2679 a268 a2680 a2681 a2682 a2683 a2684 a2685 a2686 a2687 a2688 a2689 a269 a2690 a2691 a2692 a2693 a2694 a2695 a2696 a2697 a2698 a2699 a27 a270 a2700 a2701 a2702 a2703 a2704 a2705 a2706 a2707 a2708 a2709 a271 a2710 a2711 a2712 a2713 a2714 a2715 a2716 a2717 a2718 a2719 a272 a2720 a2721 a2722 a2723 a2724 a2725 a2726 a2727 a2728 a2729 a273 a2730 a2731 a2732 a2733 a2734 a2735 a2736 a2737 a2738 a2739 a274 a2740 a2741 a2742 a2743 a2744 a2745 a2746 a2747 a2748 a2749 a275 a2750 a2751 a2752 a2753 a2754 a2755 a2756 a2757 a2758 a2759 a276 a2760 a2761 a2762 a2763 a2764 a2765 a2766 a2767 a2768 a2769 a277 a2770 a2771 a2772 a2773 a2774 a2775 a2776 a2777 a2778 a2779 a278 a2780 a2781 a2782 a2783 a2784 a2785 a2786 a2787 a2788 a2789 a279 a2790 a2791 a2792 a2793 a2794 a2795 a2796 a2797 a2798 a2799 a28 a280 a2800 a2801 a2802 a2803 a2804 a2805 a2806 a2807 a2808 a2809 a281 a2810 a2811 a2812 a2813 a2814 a2815 a2816 a2817 a2818 a2819 a282 a2820 a2821 a2822 a2823 a2824 a2825 a2826 a2827 a2828 a2829 a283 a2830 a2831 a2832 a2833 a2834 a2835 a2836 a2837 a2838 a2839 a284 a2840 a2841 a2842 a2843 a2844 a2845 a2846 a2847 a2848 a2849 a285 a2850 a2851 a2852 a2853 a2854 a2855 a2856 a2857 a2858 a2859 a286 a2860 a2861 a2862 a2863 a2864 a2865 a2866 a2867 a2868 a2869 a287 a2870 a2871 a2872 a2873 a2874 a2875 a2876 a2877 a2878 a2879 a288 a2880 a2881 a2882 a2883 a2884 a2885 a2886 a2887 a2888 a2889 a289 a2890 a2891 a2892 a2893 a2894 a2895 a2896 a2897 a2898 a2899 a29 a290 a2900 a2901 a2902 a2903 a2904 a2905 a2906 a2907 a2908 a2909 a291 a2910 a2911 a2912 a2913 a2914 a2915 a2916 a2917 a2918 a2919 a292 a2920 a2921 a2922 a2923 a2924 a2925 a2926 a2927 a2928 a2929 a293 a2930 a2931 a2932 a2933 a2934 a2935 a2936 a2937 a2938 a2939 a294 a2940 a2941 a2942 a2943 a2944 a2945 a2946 a2947 a2948 a2949 a295 a2950 a2951 a2952 a2953 a2954 a2955 a2956 a2957 a2958 a2959 a296 a2960 a2961 a2962 a2963 a2964 a2965 a2966 a2967 a2968 a2969 a297 a2970 a2971 a2972 a2973 a2974 a2975 a2976 a2977 a2978 a2979 a298 a2980 a2981 a2982 a2983 a2984 a2985 a2986 a2987 a2988 a2989 a299 a2990 a2991 a2992 a2993 a2994 a2995 a2996 a2997 a2998 a2999 a3 a30 a300 a3000 a3001 a3002 a3003 a3004 a3005 a3006 a3007 a3008 a3009 a301 a3010 a3011 a3012 a3013 a3014 a3015 a3016 a3017 a3018 a3019 a302 a3020 a3021 a3022 a3023 a3024 a3025 a3026 a3027 a3028 a3029 a303 a3030 a3031 a3032 a3033 a3034 a3035 a3036 a3037 a3038 a3039 a304 a3040 a3041 a3042 a3043 a3044 a3045 a3046 a3047 a3048 a3049 a305 a3050 a3051 a3052 a3053 a3054 a3055 a3056 a3057 a3058 a3059 a306 a3060 a3061 a3062 a3063 a3064 a3065 a3066 a3067 a3068 a3069 a307 a3070 a3071 a3072 a3073 a3074 a3075 a3076 a3077 a3078 a3079 a308 a3080 a3081 a3082 a3083 a3084 a3085 a3086 a3087 a3088 a3089 a309 a3090 a3091 a3092 a3093 a3094 a3095 a3096 a3097 a3098 a3099 a31 a310 a3100 a3101 a3102 a3103 a3104 a3105 a3106 a3107 a3108 a3109 a311 a3110 a3111 a3112 a3113 a3114 a3115 a3116 a3117 a3118 a3119 a312 a3120 a3121 a3122 a3123 a3124 a3125 a3126 a3127 a3128 a3129 a313 a3130 a3131 a3132 a3133 a3134 a3135 a3136 a3137 a3138 a3139 a314 a3140 a3141 a3142 a3143 a3144 a3145 a3146 a3147 a3148 a3149 a315 a3150 a3151 a3152 a3153 a3154 a3155 a3156 a3157 a3158 a3159 a316 a3160 a3161 a3162 a3163 a3164 a3165 a3166 a3167 a3168 a3169 a317 a3170 a3171 a3172 a3173 a3174 a3175 a3176 a3177 a3178 a3179 a318 a3180 a3181 a3182 a3183 a3184 a3185 a3186 a3187 a3188 a3189 a319 a3190 a3191 a3192 a3193 a3194 a3195 a3196 a3197 a3198 a3199 a32 a320 a3200 a3201 a3202 a3203 a3204 a3205 a3206 a3207 a3208 a3209 a321 a3210 a3211 a3212 a3213 a3214 a3215 a3216 a3217 a3218 a3219 a322 a3220 a3221 a3222 a3223 a3224 a3225 a3226 a3227 a3228 a3229 a323 a3230 a3231 a3232 a3233 a3234 a3235 a3236 a3237 a3238 a3239 a324 a3240 a3241 a3242 a3243 a3244 a3245 a3246 a3247 a3248 a3249 a325 a3250 a3251 a3252 a3253 a3254 a3255 a3256 a3257 a3258 a3259 a326 a3260 a3261 a3262 a3263 a3264 a3265 a3266 a3267 a3268 a3269 a327 a3270 a3271 a3272 a3273 a3274 a3275 a3276 a3277 a3278 a3279 a328 a3280 a3281 a3282 a3283 a3284 a3285 a3286 a3287 a3288 a3289 a329 a3290 a3291 a3292 a3293 a3294 a3295 a3296 a3297 a3298 a3299 a33 a330 a3300 a3301 a3302 a3303 a3304 a3305 a3306 a3307 a3308 a3309 a331 a3310 a3311 a3312 a3313 a3314 a3315 a3316 a3317 a3318 a3319 a332 a3320 a3321 a3322 a3323 a3324 a3325 a3326 a3327 a3328 a3329 a333 a3330 a3331 a3332 a3333 a3334 a3335 a3336 a3337 a3338 a3339 a334 a3340 a3341 a3342 a3343 a3344 a3345 a3346 a3347 a3348 a3349 a335 a3350 a3351 a3352 a3353 a3354 a3355 a3356 a3357 a3358 a3359 a336 a3360 a3361 a3362 a3363 a3364 a3365 a3366 a3367 a3368 a3369 a337 a3370 a3371 a3372 a3373 a3374 a3375 a3376 a3377 a3378 a3379 a338 a3380 a3381 a3382 a3383 a3384 a3385 a3386 a3387 a3388 a3389 a339 a3390 a3391 a3392 a3393 a3394 a3395 a3396 a3397 a3398 a3399 a34 a340 a3400 a3401 a3402 a3403 a3404 a3405 a3406 a3407 a3408 a3409 a341 a3410 a3411 a3412 a3413 a3414 a3415 a3416 a3417 a3418 a3419 a342 a3420 a3421 a3422 a3423 a3424 a3425 a3426 a3427 a3428 a3429 a343 a3430 a3431 a3432 a3433 a3434 a3435 a3436 a3437 a3438 a3439 a344 a3440 a3441 a3442 a3443 a3444 a3445 a3446 a3447 a3448 a3449 a345 a3450 a3451 a3452 a3453 a3454 a3455 a3456 a3457 a3458 a3459 a346 a3460 a3461 a3462 a3463 a3464 a3465 a3466 a3467 a3468 a3469 a347 a3470 a3471 a3472 a3473 a3474 a3475 a3476 a3477 a3478 a3479 a348 a3480 a3481 a3482 a3483 a3484 a3485 a3486 a3487 a3488 a3489 a349 a3490 a3491 a3492 a3493 a3494 a3495 a3496 a3497 a3498 a3499 a35 a350 a3500 a3501 a3502 a3503 a3504 a3505 a3506 a3507 a3508 a3509 a351 a3510 a3511 a3512 a3513 a3514 a3515 a3516 a3517 a3518 a3519 a352 a3520 a3521 a3522 a3523 a3524 a3525 a3526 a3527 a3528 a3529 a353 a3530 a3531 a3532 a3533 a3534 a3535 a3536 a3537 a3538 a3539 a354 a3540 a3541 a3542 a3543 a3544 a3545 a3546 a3547 a3548 a3549 a355 a3550 a3551 a3552 a3553 a3554 a3555 a3556 a3557 a3558 a3559 a356 a3560 a3561 a3562 a3563 a3564 a3565 a3566 a3567 a3568 a3569 a357 a3570 a3571 a3572 a3573 a3574 a3575 a3576 a3577 a3578 a3579 a358 a3580 a3581 a3582 a3583 a3584 a3585 a3586 a3587 a3588 a3589 a359 a3590 a3591 a3592 a3593 a3594 a3595 a3596 a3597 a3598 a3599 a36 a360 a3600 a3601 a3602 a3603 a3604 a3605 a3606 a3607 a3608 a3609 a361 a3610 a3611 a3612 a3613 a3614 a3615 a3616 a3617 a3618 a3619 a362 a3620 a3621 a3622 a3623 a3624 a3625 a3626 a3627 a3628 a3629 a363 a3630 a3631 a3632 a3633 a3634 a3635 a3636 a3637 a3638 a3639 a364 a3640 a3641 a3642 a3643 a3644 a3645 a3646 a3647 a3648 a3649 a365 a3650 a3651 a3652 a3653 a3654 a3655 a3656 a3657 a3658 a3659 a366 a3660 a3661 a3662 a3663 a3664 a3665 a3666 a3667 a3668 a3669 a367 a3670 a3671 a3672 a3673 a3674 a3675 a3676 a3677 a3678 a3679 a368 a3680 a3681 a3682 a3683 a3684 a3685 a3686 a3687 a3688 a3689 a369 a3690 a3691 a3692 a3693 a3694 a3695 a3696 a3697 a3698 a3699 a37 a370 a3700 a3701 a3702 a3703 a3704 a3705 a3706 a3707 a3708 a3709 a371 a3710 a3711 a3712 a3713 a3714 a3715 a3716 a3717 a3718 a3719 a372 a3720 a3721 a3722 a3723 a3724 a3725 a3726 a3727 a3728 a3729 a373 a3730 a3731 a3732 a3733 a3734 a3735 a3736 a3737 a3738 a3739 a374 a3740 a3741 a3742 a3743 a3744 a3745 a3746 a3747 a3748 a3749 a375 a3750 a3751 a3752 a3753 a3754 a3755 a3756 a3757 a3758 a3759 a376 a3760 a3761 a3762 a3763 a3764 a3765 a3766 a3767 a3768 a3769 a377 a3770 a3771 a3772 a3773 a3774 a3775 a3776 a3777 a3778 a3779 a378 a3780 a3781 a3782 a3783 a3784 a3785 a3786 a3787 a3788 a3789 a379 a3790 a3791 a3792 a3793 a3794 a3795 a3796 a3797 a3798 a3799 a38 a380 a3800 a3801 a3802 a3803 a3804 a3805 a3806 a3807 a3808 a3809 a381 a3810 a3811 a3812 a3813 a3814 a3815 a3816 a3817 a3818 a3819 a382 a3820 a3821 a3822 a3823 a3824 a3825 a3826 a3827 a3828 a3829 a383 a3830 a3831 a3832 a3833 a3834 a3835 a3836 a3837 a3838 a3839 a384 a3840 a3841 a3842 a3843 a3844 a3845 a3846 a3847 a3848 a3849 a385 a3850 a3851 a3852 a3853 a3854 a3855 a3856 a3857 a3858 a3859 a386 a3860 a3861 a3862 a3863 a3864 a3865 a3866 a3867 a3868 a3869 a387 a3870 a3871 a3872 a3873 a3874 a3875 a3876 a3877 a3878 a3879 a388 a3880 a3881 a3882 a3883 a3884 a3885 a3886 a3887 a3888 a3889 a389 a3890 a3891 a3892 a3893 a3894 a3895 a3896 a3897 a3898 a3899 a39 a390 a3900 a3901 a3902 a3903 a3904 a3905 a3906 a3907 a3908 a3909 a391 a3910 a3911 a3912 a3913 a3914 a3915 a3916 a3917 a3918 a3919 a392 a3920 a3921 a3922 a3923 a3924 a3925 a3926 a3927 a3928 a3929 a393 a3930 a3931 a3932 a3933 a3934 a3935 a3936 a3937 a3938 a3939 a394 a3940 a3941 a3942 a3943 a3944 a3945 a3946 a3947 a3948 a3949 a395 a3950 a3951 a3952 a3953 a3954 a3955 a3956 a3957 a3958 a3959 a396 a3960 a3961 a3962 a3963 a3964 a3965 a3966 a3967 a3968 a3969 a397 a3970 a3971 a3972 a3973 a3974 a3975 a3976 a3977 a3978 a3979 a398 a3980 a3981 a3982 a3983 a3984 a3985 a3986 a3987 a3988 a3989 a399 a3990 a3991 a3992 a3993 a3994 a3995 a3996 a3997 a3998 a3999 a4 a40 a400 a401 a402 a403 a404 a405 a406 a407 a408 a409 a41 a410 a411 a412 a413 a414 a415 a416 a417 a418 a419 a42 a420 a421 a422 a423 a424 a425 a426 a427 a428 a429 a43 a430 a431 a432 a433 a434 a435 a436 a437 a438 a439 a44 a440 a441 a442 a443 a444 a445 a446 a447 a448 a449 a45 a450 a451 a452 a453 a454 a455 a456 a457 a458 a459 a46 a460 a461 a462 a463 a464 a465 a466 a467 a468 a469 a47 a470 a471 a472 a473 a474 a475 a476 a477 a478 a479 a48 a480 a481 a482 a483 a484 a485 a486 a487 a488 a489 a49 a490 a491 a492 a493 a494 a495 a496 a497 a498 a499 a5 a50 a500 a501 a502 a503 a504 a505 a506 a507 a508 a509 a51 a510 a511 a512 a513 a514 a515 a516 a517 a518 a519 a52 a520 a521 a522 a523 a524 a525 a526 a527 a528 a529 a53 a530 a531 a532 a533 a534 a535 a536 a537 a538 a539 a54 a540 a541 a542 a543 a544 a545 a546 a547 a548 a549 a55 a550 a551 a552 a553 a554 a555 a556 a557 a558 a559 a56 a560 a561 a562 a563 a564 a565 a566 a567 a568 a569 a57 a570 a571 a572 a573 a574 a575 a576 a577 a578 a579 a58 a580 a581 a582 a583 a584 a585 a586 a587 a588 a589 a59 a590 a591 a592 a593 a594 a595 a596 a597 a598 a599 a6 a60 a600 a601 a602 a603 a604 a605 a606 a607 a608 a609 a61 a610 a611 a612 a613 a614 a615 a616 a617 a618 a619 a62 a620 a621 a622 a623 a624 a625 a626 a627 a628 a629 a63 a630 a631 a632 a633 a634 a635 a636 a637 a638 a639 a64 a640 a641 a642 a643 a644 a645 a646 a647 a648 a649 a65 a650 a651 a652 a653 a654 a655 a656 a657 a658 a659 a66 a660 a661 a662 a663 a664 a665 a666 a667 a668 a669 a67 a670 a671 a672 a673 a674 a675 a676 a677 a678 a679 a68 a680 a681 a682 a683 a684 a685 a686 a687 a688 a689 a69 a690 a691 a692 a693 a694 a695 a696 a697 a698 a699 a7 a70 a700 a701 a702 a703 a704 a705 a706 a707 a708 a709 a71 a710 a711 a712 a713 a714 a715 a716 a717 a718 a719 a72 a720 a721 a722 a723 a724 a725 a726 a727 a728 a729 a73 a730 a731 a732 a733 a734 a735 a736 a737 a738 a739 a74 a740 a741 a742 a743 a744 a745 a746 a747 a748 a749 a75 a750 a751 a752 a753 a754 a755 a756 a757 a758 a759 a76 a760 a761 a762 a763 a764 a765 a766 a767 a768 a769 a77 a770 a771 a772 a773 a774 a775 a776 a777 a778 a779 a78 a780 a781 a782 a783 a784 a785 a786 a787 a788 a789 a79 a790 a791 a792 a793 a794 a795 a796 a797 a798 a799 a8 a80 a800 a801 a802 a803 a804 a805 a806 a807 a808 a809 a81 a810 a811 a812 a813 a814 a815 a816 a817 a818 a819 a82 a820 a821 a822 a823 a824 a825 a826 a827 a828 a829 a83 a830 a831 a832 a833 a834 a835 a836 a837 a838 a839 a84 a840 a841 a842 a843 a844 a845 a846 a847 a848 a849 a85 a850 a851 a852 a853 a854 a855 a856 a857 a858 a859 a86 a860 a861 a862 a863 a864 a865 a866 a867 a868 a869 a87 a870 a871 a872 a873 a874 a875 a876 a877 a878 a879 a88 a880 a881 a882 a883 a884 a885 a886 a887 a888 a889 a89 a890 a891 a892 a893 a894 a895 a896 a897 a898 a899 a9 a90 a900 a901 a902 a903 a904 a905 a906 a907 a908 a909 a91 a910 a911 a912 a913 a914 a915 a916 a917 a918 a919 a92 a920 a921 a922 a923 a924 a925 a926 a927 a928 a929 a93 a930 a931 a932 a933 a934 a935 a936 a937 a938 a939 a94 a940 a941 a942 a943 a944 a945 a946 a947 a948 a949 a95 a950 a951 a952 a953 a954 a955 a956 a957 a958 a959 a96 a960 a961 a962 a963 a964 a965 a966 a967 a968 a969 a97 a970 a971 a972 a973 a974 a975 a976 a977 a978 a979 a98 a980 a981 a982 a983 a984 a985 a986 a987 a988 a989 a99 a990 a991 a992 a993 a994 a995 a996 a997 a998 a999 ./d36.sanityn: ./d37.sanityn: f0 f1 f10 f100 f1000 f1001 f1002 f1003 f1004 f1005 f1006 f1007 f1008 f1009 f101 f1010 f1011 f1012 f1013 f1014 f1015 f1016 f1017 f1018 f1019 f102 f1020 f1021 f1022 f1023 f1024 f1025 f1026 f1027 f1028 f1029 f103 f1030 f1031 f1032 f1033 f1034 f1035 f1036 f1037 f1038 f1039 f104 f1040 f1041 f1042 f1043 f1044 f1045 f1046 f1047 f1048 f1049 f105 f1050 f1051 f1052 f1053 f1054 f1055 f1056 f1057 f1058 f1059 f106 f1060 f1061 f1062 f1063 f1064 f1065 f1066 f1067 f1068 f1069 f107 f1070 f1071 f1072 f1073 f1074 f1075 f1076 f1077 f1078 f1079 f108 f1080 f1081 f1082 f1083 f1084 f1085 f1086 f1087 f1088 f1089 f109 f1090 f1091 f1092 f1093 f1094 f1095 f1096 f1097 f1098 f1099 f11 f110 f1100 f1101 f1102 f1103 f1104 f1105 f1106 f1107 f1108 f1109 f111 f1110 f1111 f1112 f1113 f1114 f1115 f1116 f1117 f1118 f1119 f112 f1120 f1121 f1122 f1123 f1124 f1125 f1126 f1127 f1128 f1129 f113 f1130 f1131 f1132 f1133 f1134 f1135 f1136 f1137 f1138 f1139 f114 f1140 f1141 f1142 f1143 f1144 f1145 f1146 f1147 f1148 f1149 f115 f1150 f1151 f1152 f1153 f1154 f1155 f1156 f1157 f1158 f1159 f116 f1160 f1161 f1162 f1163 f1164 f1165 f1166 f1167 f1168 f1169 f117 f1170 f1171 f1172 f1173 f1174 f1175 f1176 f1177 f1178 f1179 f118 f1180 f1181 f1182 f1183 f1184 f1185 f1186 f1187 f1188 f1189 f119 f1190 f1191 f1192 f1193 f1194 f1195 f1196 f1197 f1198 f1199 f12 f120 f1200 f1201 f1202 f1203 f1204 f1205 f1206 f1207 f1208 f1209 f121 f1210 f1211 f1212 f1213 f1214 f1215 f1216 f1217 f1218 f1219 f122 f1220 f1221 f1222 f1223 f1224 f1225 f1226 f1227 f1228 f1229 f123 f1230 f1231 f1232 f1233 f1234 f1235 f1236 f1237 f1238 f1239 f124 f1240 f1241 f1242 f1243 f1244 f1245 f1246 f1247 f1248 f1249 f125 f1250 f1251 f1252 f1253 f1254 f1255 f1256 f1257 f1258 f1259 f126 f1260 f1261 f1262 f1263 f1264 f1265 f1266 f1267 f1268 f1269 f127 f1270 f1271 f1272 f1273 f1274 f1275 f1276 f1277 f1278 f1279 f128 f1280 f1281 f1282 f1283 f1284 f1285 f1286 f1287 f1288 f1289 f129 f1290 f1291 f1292 f1293 f1294 f1295 f1296 f1297 f1298 f1299 f13 f130 f1300 f1301 f1302 f1303 f1304 f1305 f1306 f1307 f1308 f1309 f131 f1310 f1311 f1312 f1313 f1314 f1315 f1316 f1317 f1318 f1319 f132 f1320 f1321 f1322 f1323 f1324 f1325 f1326 f1327 f1328 f1329 f133 f1330 f1331 f1332 f1333 f1334 f1335 f1336 f1337 f1338 f1339 f134 f1340 f1341 f1342 f1343 f1344 f1345 f1346 f1347 f1348 f1349 f135 f1350 f1351 f1352 f1353 f1354 f1355 f1356 f1357 f1358 f1359 f136 f1360 f1361 f1362 f1363 f1364 f1365 f1366 f1367 f1368 f1369 f137 f1370 f1371 f1372 f1373 f1374 f1375 f1376 f1377 f1378 f1379 f138 f1380 f1381 f1382 f1383 f1384 f1385 f1386 f1387 f1388 f1389 f139 f1390 f1391 f1392 f1393 f1394 f1395 f1396 f1397 f1398 f1399 f14 f140 f1400 f1401 f1402 f1403 f1404 f1405 f1406 f1407 f1408 f1409 f141 f1410 f1411 f1412 f1413 f1414 f1415 f1416 f1417 f1418 f1419 f142 f1420 f1421 f1422 f1423 f1424 f1425 f1426 f1427 f1428 f1429 f143 f1430 f1431 f1432 f1433 f1434 f1435 f1436 f1437 f1438 f1439 f144 f1440 f1441 f1442 f1443 f1444 f1445 f1446 f1447 f1448 f1449 f145 f1450 f1451 f1452 f1453 f1454 f1455 f1456 f1457 f1458 f1459 f146 f1460 f1461 f1462 f1463 f1464 f1465 f1466 f1467 f1468 f1469 f147 f1470 f1471 f1472 f1473 f1474 f1475 f1476 f1477 f1478 f1479 f148 f1480 f1481 f1482 f1483 f1484 f1485 f1486 f1487 f1488 f1489 f149 f1490 f1491 f1492 f1493 f1494 f1495 f1496 f1497 f1498 f1499 f15 f150 f1500 f1501 f1502 f1503 f1504 f1505 f1506 f1507 f1508 f1509 f151 f1510 f1511 f1512 f1513 f1514 f1515 f1516 f1517 f1518 f1519 f152 f1520 f1521 f1522 f1523 f1524 f1525 f1526 f1527 f1528 f1529 f153 f1530 f1531 f1532 f1533 f1534 f1535 f1536 f1537 f1538 f1539 f154 f1540 f1541 f1542 f1543 f1544 f1545 f1546 f1547 f1548 f1549 f155 f1550 f1551 f1552 f1553 f1554 f1555 f1556 f1557 f1558 f1559 f156 f1560 f1561 f1562 f1563 f1564 f1565 f1566 f1567 f1568 f1569 f157 f1570 f1571 f1572 f1573 f1574 f1575 f1576 f1577 f1578 f1579 f158 f1580 f1581 f1582 f1583 f1584 f1585 f1586 f1587 f1588 f1589 f159 f1590 f1591 f1592 f1593 f1594 f1595 f1596 f1597 f1598 f1599 f16 f160 f1600 f1601 f1602 f1603 f1604 f1605 f1606 f1607 f1608 f1609 f161 f1610 f1611 f1612 f1613 f1614 f1615 f1616 f1617 f1618 f1619 f162 f1620 f1621 f1622 f1623 f1624 f1625 f1626 f1627 f1628 f1629 f163 f1630 f1631 f1632 f1633 f1634 f1635 f1636 f1637 f1638 f1639 f164 f1640 f1641 f1642 f1643 f1644 f1645 f1646 f1647 f1648 f1649 f165 f1650 f1651 f1652 f1653 f1654 f1655 f1656 f1657 f1658 f1659 f166 f1660 f1661 f1662 f1663 f1664 f1665 f1666 f1667 f1668 f1669 f167 f1670 f1671 f1672 f1673 f1674 f1675 f1676 f1677 f1678 f1679 f168 f1680 f1681 f1682 f1683 f1684 f1685 f1686 f1687 f1688 f1689 f169 f1690 f1691 f1692 f1693 f1694 f1695 f1696 f1697 f1698 f1699 f17 f170 f1700 f1701 f1702 f1703 f1704 f1705 f1706 f1707 f1708 f1709 f171 f1710 f1711 f1712 f1713 f1714 f1715 f1716 f1717 f1718 f1719 f172 f1720 f1721 f1722 f1723 f1724 f1725 f1726 f1727 f1728 f1729 f173 f1730 f1731 f1732 f1733 f1734 f1735 f1736 f1737 f1738 f1739 f174 f1740 f1741 f1742 f1743 f1744 f1745 f1746 f1747 f1748 f1749 f175 f1750 f1751 f1752 f1753 f1754 f1755 f1756 f1757 f1758 f1759 f176 f1760 f1761 f1762 f1763 f1764 f1765 f1766 f1767 f1768 f1769 f177 f1770 f1771 f1772 f1773 f1774 f1775 f1776 f1777 f1778 f1779 f178 f1780 f1781 f1782 f1783 f1784 f1785 f1786 f1787 f1788 f1789 f179 f1790 f1791 f1792 f1793 f1794 f1795 f1796 f1797 f1798 f1799 f18 f180 f1800 f1801 f1802 f1803 f1804 f1805 f1806 f1807 f1808 f1809 f181 f1810 f1811 f1812 f1813 f1814 f1815 f1816 f1817 f1818 f1819 f182 f1820 f1821 f1822 f1823 f1824 f1825 f1826 f1827 f1828 f1829 f183 f1830 f1831 f1832 f1833 f1834 f1835 f1836 f1837 f1838 f1839 f184 f1840 f1841 f1842 f1843 f1844 f1845 f1846 f1847 f1848 f1849 f185 f1850 f1851 f1852 f1853 f1854 f1855 f1856 f1857 f1858 f1859 f186 f1860 f1861 f1862 f1863 f1864 f1865 f1866 f1867 f1868 f1869 f187 f1870 f1871 f1872 f1873 f1874 f1875 f1876 f1877 f1878 f1879 f188 f1880 f1881 f1882 f1883 f1884 f1885 f1886 f1887 f1888 f1889 f189 f1890 f1891 f1892 f1893 f1894 f1895 f1896 f1897 f1898 f1899 f19 f190 f1900 f1901 f1902 f1903 f1904 f1905 f1906 f1907 f1908 f1909 f191 f1910 f1911 f1912 f1913 f1914 f1915 f1916 f1917 f1918 f1919 f192 f1920 f1921 f1922 f1923 f1924 f1925 f1926 f1927 f1928 f1929 f193 f1930 f1931 f1932 f1933 f1934 f1935 f1936 f1937 f1938 f1939 f194 f1940 f1941 f1942 f1943 f1944 f1945 f1946 f1947 f1948 f1949 f195 f1950 f1951 f1952 f1953 f1954 f1955 f1956 f1957 f1958 f1959 f196 f1960 f1961 f1962 f1963 f1964 f1965 f1966 f1967 f1968 f1969 f197 f1970 f1971 f1972 f1973 f1974 f1975 f1976 f1977 f1978 f1979 f198 f1980 f1981 f1982 f1983 f1984 f1985 f1986 f1987 f1988 f1989 f199 f1990 f1991 f1992 f1993 f1994 f1995 f1996 f1997 f1998 f1999 f2 f20 f200 f2000 f2001 f2002 f2003 f2004 f2005 f2006 f2007 f2008 f2009 f201 f2010 f2011 f2012 f2013 f2014 f2015 f2016 f2017 f2018 f2019 f202 f2020 f2021 f2022 f2023 f2024 f2025 f2026 f2027 f2028 f2029 f203 f2030 f2031 f2032 f2033 f2034 f2035 f2036 f2037 f2038 f2039 f204 f2040 f2041 f2042 f2043 f2044 f2045 f2046 f2047 f2048 f2049 f205 f2050 f2051 f2052 f2053 f2054 f2055 f2056 f2057 f2058 f2059 f206 f2060 f2061 f2062 f2063 f2064 f2065 f2066 f2067 f2068 f2069 f207 f2070 f2071 f2072 f2073 f2074 f2075 f2076 f2077 f2078 f2079 f208 f2080 f2081 f2082 f2083 f2084 f2085 f2086 f2087 f2088 f2089 f209 f2090 f2091 f2092 f2093 f2094 f2095 f2096 f2097 f2098 f2099 f21 f210 f2100 f2101 f2102 f2103 f2104 f2105 f2106 f2107 f2108 f2109 f211 f2110 f2111 f2112 f2113 f2114 f2115 f2116 f2117 f2118 f2119 f212 f2120 f2121 f2122 f2123 f2124 f2125 f2126 f2127 f2128 f2129 f213 f2130 f2131 f2132 f2133 f2134 f2135 f2136 f2137 f2138 f2139 f214 f2140 f2141 f2142 f2143 f2144 f2145 f2146 f2147 f2148 f2149 f215 f2150 f2151 f2152 f2153 f2154 f2155 f2156 f2157 f2158 f2159 f216 f2160 f2161 f2162 f2163 f2164 f2165 f2166 f2167 f2168 f2169 f217 f2170 f2171 f2172 f2173 f2174 f2175 f2176 f2177 f2178 f2179 f218 f2180 f2181 f2182 f2183 f2184 f2185 f2186 f2187 f2188 f2189 f219 f2190 f2191 f2192 f2193 f2194 f2195 f2196 f2197 f2198 f2199 f22 f220 f2200 f2201 f2202 f2203 f2204 f2205 f2206 f2207 f2208 f2209 f221 f2210 f2211 f2212 f2213 f2214 f2215 f2216 f2217 f2218 f2219 f222 f2220 f2221 f2222 f2223 f2224 f2225 f2226 f2227 f2228 f2229 f223 f2230 f2231 f2232 f2233 f2234 f2235 f2236 f2237 f2238 f2239 f224 f2240 f2241 f2242 f2243 f2244 f2245 f2246 f2247 f2248 f2249 f225 f2250 f2251 f2252 f2253 f2254 f2255 f2256 f2257 f2258 f2259 f226 f2260 f2261 f2262 f2263 f2264 f2265 f2266 f2267 f2268 f2269 f227 f2270 f2271 f2272 f2273 f2274 f2275 f2276 f2277 f2278 f2279 f228 f2280 f2281 f2282 f2283 f2284 f2285 f2286 f2287 f2288 f2289 f229 f2290 f2291 f2292 f2293 f2294 f2295 f2296 f2297 f2298 f2299 f23 f230 f2300 f2301 f2302 f2303 f2304 f2305 f2306 f2307 f2308 f2309 f231 f2310 f2311 f2312 f2313 f2314 f2315 f2316 f2317 f2318 f2319 f232 f2320 f2321 f2322 f2323 f2324 f2325 f2326 f2327 f2328 f2329 f233 f2330 f2331 f2332 f2333 f2334 f2335 f2336 f2337 f2338 f2339 f234 f2340 f2341 f2342 f2343 f2344 f2345 f2346 f2347 f2348 f2349 f235 f2350 f2351 f2352 f2353 f2354 f2355 f2356 f2357 f2358 f2359 f236 f2360 f2361 f2362 f2363 f2364 f2365 f2366 f2367 f2368 f2369 f237 f2370 f2371 f2372 f2373 f2374 f2375 f2376 f2377 f2378 f2379 f238 f2380 f2381 f2382 f2383 f2384 f2385 f2386 f2387 f2388 f2389 f239 f2390 f2391 f2392 f2393 f2394 f2395 f2396 f2397 f2398 f2399 f24 f240 f2400 f2401 f2402 f2403 f2404 f2405 f2406 f2407 f2408 f2409 f241 f2410 f2411 f2412 f2413 f2414 f2415 f2416 f2417 f2418 f2419 f242 f2420 f2421 f2422 f2423 f2424 f2425 f2426 f2427 f2428 f2429 f243 f2430 f2431 f2432 f2433 f2434 f2435 f2436 f2437 f2438 f2439 f244 f2440 f2441 f2442 f2443 f2444 f2445 f2446 f2447 f2448 f2449 f245 f2450 f2451 f2452 f2453 f2454 f2455 f2456 f2457 f2458 f2459 f246 f2460 f2461 f2462 f2463 f2464 f2465 f2466 f2467 f2468 f2469 f247 f2470 f2471 f2472 f2473 f2474 f2475 f2476 f2477 f2478 f2479 f248 f2480 f2481 f2482 f2483 f2484 f2485 f2486 f2487 f2488 f2489 f249 f2490 f2491 f2492 f2493 f2494 f2495 f2496 f2497 f2498 f2499 f25 f250 f2500 f2501 f2502 f2503 f2504 f2505 f2506 f2507 f2508 f2509 f251 f2510 f2511 f2512 f2513 f2514 f2515 f2516 f2517 f2518 f2519 f252 f2520 f2521 f2522 f2523 f2524 f2525 f2526 f2527 f2528 f2529 f253 f2530 f2531 f2532 f2533 f2534 f2535 f2536 f2537 f2538 f2539 f254 f2540 f2541 f2542 f2543 f2544 f2545 f2546 f2547 f2548 f2549 f255 f2550 f2551 f2552 f2553 f2554 f2555 f2556 f2557 f2558 f2559 f256 f2560 f2561 f2562 f2563 f2564 f2565 f2566 f2567 f2568 f2569 f257 f2570 f2571 f2572 f2573 f2574 f2575 f2576 f2577 f2578 f2579 f258 f2580 f2581 f2582 f2583 f2584 f2585 f2586 f2587 f2588 f2589 f259 f2590 f2591 f2592 f2593 f2594 f2595 f2596 f2597 f2598 f2599 f26 f260 f2600 f2601 f2602 f2603 f2604 f2605 f2606 f2607 f2608 f2609 f261 f2610 f2611 f2612 f2613 f2614 f2615 f2616 f2617 f2618 f2619 f262 f2620 f2621 f2622 f2623 f2624 f2625 f2626 f2627 f2628 f2629 f263 f2630 f2631 f2632 f2633 f2634 f2635 f2636 f2637 f2638 f2639 f264 f2640 f2641 f2642 f2643 f2644 f2645 f2646 f2647 f2648 f2649 f265 f2650 f2651 f2652 f2653 f2654 f2655 f2656 f2657 f2658 f2659 f266 f2660 f2661 f2662 f2663 f2664 f2665 f2666 f2667 f2668 f2669 f267 f2670 f2671 f2672 f2673 f2674 f2675 f2676 f2677 f2678 f2679 f268 f2680 f2681 f2682 f2683 f2684 f2685 f2686 f2687 f2688 f2689 f269 f2690 f2691 f2692 f2693 f2694 f2695 f2696 f2697 f2698 f2699 f27 f270 f2700 f2701 f2702 f2703 f2704 f2705 f2706 f2707 f2708 f2709 f271 f2710 f2711 f2712 f2713 f2714 f2715 f2716 f2717 f2718 f2719 f272 f2720 f2721 f2722 f2723 f2724 f2725 f2726 f2727 f2728 f2729 f273 f2730 f2731 f2732 f2733 f2734 f2735 f2736 f2737 f2738 f2739 f274 f2740 f2741 f2742 f2743 f2744 f2745 f2746 f2747 f2748 f2749 f275 f2750 f2751 f2752 f2753 f2754 f2755 f2756 f2757 f2758 f2759 f276 f2760 f2761 f2762 f2763 f2764 f2765 f2766 f2767 f2768 f2769 f277 f2770 f2771 f2772 f2773 f2774 f2775 f2776 f2777 f2778 f2779 f278 f2780 f2781 f2782 f2783 f2784 f2785 f2786 f2787 f2788 f2789 f279 f2790 f2791 f2792 f2793 f2794 f2795 f2796 f2797 f2798 f2799 f28 f280 f2800 f2801 f2802 f2803 f2804 f2805 f2806 f2807 f2808 f2809 f281 f2810 f2811 f2812 f2813 f2814 f2815 f2816 f2817 f2818 f2819 f282 f2820 f2821 f2822 f2823 f2824 f2825 f2826 f2827 f2828 f2829 f283 f2830 f2831 f2832 f2833 f2834 f2835 f2836 f2837 f2838 f2839 f284 f2840 f2841 f2842 f2843 f2844 f2845 f2846 f2847 f2848 f2849 f285 f2850 f2851 f2852 f2853 f2854 f2855 f2856 f2857 f2858 f2859 f286 f2860 f2861 f2862 f2863 f2864 f2865 f2866 f2867 f2868 f2869 f287 f2870 f2871 f2872 f2873 f2874 f2875 f2876 f2877 f2878 f2879 f288 f2880 f2881 f2882 f2883 f2884 f2885 f2886 f2887 f2888 f2889 f289 f2890 f2891 f2892 f2893 f2894 f2895 f2896 f2897 f2898 f2899 f29 f290 f2900 f2901 f2902 f2903 f2904 f2905 f2906 f2907 f2908 f2909 f291 f2910 f2911 f2912 f2913 f2914 f2915 f2916 f2917 f2918 f2919 f292 f2920 f2921 f2922 f2923 f2924 f2925 f2926 f2927 f2928 f2929 f293 f2930 f2931 f2932 f2933 f2934 f2935 f2936 f2937 f2938 f2939 f294 f2940 f2941 f2942 f2943 f2944 f2945 f2946 f2947 f2948 f2949 f295 f2950 f2951 f2952 f2953 f2954 f2955 f2956 f2957 f2958 f2959 f296 f2960 f2961 f2962 f2963 f2964 f2965 f2966 f2967 f2968 f2969 f297 f2970 f2971 f2972 f2973 f2974 f2975 f2976 f2977 f2978 f2979 f298 f2980 f2981 f2982 f2983 f2984 f2985 f2986 f2987 f2988 f2989 f299 f2990 f2991 f2992 f2993 f2994 f2995 f2996 f2997 f2998 f2999 f3 f30 f300 f3000 f3001 f3002 f3003 f3004 f3005 f3006 f3007 f3008 f3009 f301 f3010 f3011 f3012 f3013 f3014 f3015 f3016 f3017 f3018 f3019 f302 f3020 f3021 f3022 f3023 f3024 f3025 f3026 f3027 f3028 f3029 f303 f3030 f3031 f3032 f3033 f3034 f3035 f3036 f3037 f3038 f3039 f304 f3040 f3041 f3042 f3043 f3044 f3045 f3046 f3047 f3048 f3049 f305 f3050 f3051 f3052 f3053 f3054 f3055 f3056 f3057 f3058 f3059 f306 f3060 f3061 f3062 f3063 f3064 f3065 f3066 f3067 f3068 f3069 f307 f3070 f3071 f3072 f3073 f3074 f3075 f3076 f3077 f3078 f3079 f308 f3080 f3081 f3082 f3083 f3084 f3085 f3086 f3087 f3088 f3089 f309 f3090 f3091 f3092 f3093 f3094 f3095 f3096 f3097 f3098 f3099 f31 f310 f3100 f3101 f3102 f3103 f3104 f3105 f3106 f3107 f3108 f3109 f311 f3110 f3111 f3112 f3113 f3114 f3115 f3116 f3117 f3118 f3119 f312 f3120 f3121 f3122 f3123 f3124 f3125 f3126 f3127 f3128 f3129 f313 f3130 f3131 f3132 f3133 f3134 f3135 f3136 f3137 f3138 f3139 f314 f3140 f3141 f3142 f3143 f3144 f3145 f3146 f3147 f3148 f3149 f315 f3150 f3151 f3152 f3153 f3154 f3155 f3156 f3157 f3158 f3159 f316 f3160 f3161 f3162 f3163 f3164 f3165 f3166 f3167 f3168 f3169 f317 f3170 f3171 f3172 f3173 f3174 f3175 f3176 f3177 f3178 f3179 f318 f3180 f3181 f3182 f3183 f3184 f3185 f3186 f3187 f3188 f3189 f319 f3190 f3191 f3192 f3193 f3194 f3195 f3196 f3197 f3198 f3199 f32 f320 f3200 f3201 f3202 f3203 f3204 f3205 f3206 f3207 f3208 f3209 f321 f3210 f3211 f3212 f3213 f3214 f3215 f3216 f3217 f3218 f3219 f322 f3220 f3221 f3222 f3223 f3224 f3225 f3226 f3227 f3228 f3229 f323 f3230 f3231 f3232 f3233 f3234 f3235 f3236 f3237 f3238 f3239 f324 f3240 f3241 f3242 f3243 f3244 f3245 f3246 f3247 f3248 f3249 f325 f3250 f3251 f3252 f3253 f3254 f3255 f3256 f3257 f3258 f3259 f326 f3260 f3261 f3262 f3263 f3264 f3265 f3266 f3267 f3268 f3269 f327 f3270 f3271 f3272 f3273 f3274 f3275 f3276 f3277 f3278 f3279 f328 f3280 f3281 f3282 f3283 f3284 f3285 f3286 f3287 f3288 f3289 f329 f3290 f3291 f3292 f3293 f3294 f3295 f3296 f3297 f3298 f3299 f33 f330 f3300 f3301 f3302 f3303 f3304 f3305 f3306 f3307 f3308 f3309 f331 f3310 f3311 f3312 f3313 f3314 f3315 f3316 f3317 f3318 f3319 f332 f3320 f3321 f3322 f3323 f3324 f3325 f3326 f3327 f3328 f3329 f333 f3330 f3331 f3332 f3333 f3334 f3335 f3336 f3337 f3338 f3339 f334 f3340 f3341 f3342 f3343 f3344 f3345 f3346 f3347 f3348 f3349 f335 f3350 f3351 f3352 f3353 f3354 f3355 f3356 f3357 f3358 f3359 f336 f3360 f3361 f3362 f3363 f3364 f3365 f3366 f3367 f3368 f3369 f337 f3370 f3371 f3372 f3373 f3374 f3375 f3376 f3377 f3378 f3379 f338 f3380 f3381 f3382 f3383 f3384 f3385 f3386 f3387 f3388 f3389 f339 f3390 f3391 f3392 f3393 f3394 f3395 f3396 f3397 f3398 f3399 f34 f340 f3400 f3401 f3402 f3403 f3404 f3405 f3406 f3407 f3408 f3409 f341 f3410 f3411 f3412 f3413 f3414 f3415 f3416 f3417 f3418 f3419 f342 f3420 f3421 f3422 f3423 f3424 f3425 f3426 f3427 f3428 f3429 f343 f3430 f3431 f3432 f3433 f3434 f3435 f3436 f3437 f3438 f3439 f344 f3440 f3441 f3442 f3443 f3444 f3445 f3446 f3447 f3448 f3449 f345 f3450 f3451 f3452 f3453 f3454 f3455 f3456 f3457 f3458 f3459 f346 f3460 f3461 f3462 f3463 f3464 f3465 f3466 f3467 f3468 f3469 f347 f3470 f3471 f3472 f3473 f3474 f3475 f3476 f3477 f3478 f3479 f348 f3480 f3481 f3482 f3483 f3484 f3485 f3486 f3487 f3488 f3489 f349 f3490 f3491 f3492 f3493 f3494 f3495 f3496 f3497 f3498 f3499 f35 f350 f3500 f3501 f3502 f3503 f3504 f3505 f3506 f3507 f3508 f3509 f351 f3510 f3511 f3512 f3513 f3514 f3515 f3516 f3517 f3518 f3519 f352 f3520 f3521 f3522 f3523 f3524 f3525 f3526 f3527 f3528 f3529 f353 f3530 f3531 f3532 f3533 f3534 f3535 f3536 f3537 f3538 f3539 f354 f3540 f3541 f3542 f3543 f3544 f3545 f3546 f3547 f3548 f3549 f355 f3550 f3551 f3552 f3553 f3554 f3555 f3556 f3557 f3558 f3559 f356 f3560 f3561 f3562 f3563 f3564 f3565 f3566 f3567 f3568 f3569 f357 f3570 f3571 f3572 f3573 f3574 f3575 f3576 f3577 f3578 f3579 f358 f3580 f3581 f3582 f3583 f3584 f3585 f3586 f3587 f3588 f3589 f359 f3590 f3591 f3592 f3593 f3594 f3595 f3596 f3597 f3598 f3599 f36 f360 f3600 f3601 f3602 f3603 f3604 f3605 f3606 f3607 f3608 f3609 f361 f3610 f3611 f3612 f3613 f3614 f3615 f3616 f3617 f3618 f3619 f362 f3620 f3621 f3622 f3623 f3624 f3625 f3626 f3627 f3628 f3629 f363 f3630 f3631 f3632 f3633 f3634 f3635 f3636 f3637 f3638 f3639 f364 f3640 f3641 f3642 f3643 f3644 f3645 f3646 f3647 f3648 f3649 f365 f3650 f3651 f3652 f3653 f3654 f3655 f3656 f3657 f3658 f3659 f366 f3660 f3661 f3662 f3663 f3664 f3665 f3666 f3667 f3668 f3669 f367 f3670 f3671 f3672 f3673 f3674 f3675 f3676 f3677 f3678 f3679 f368 f3680 f3681 f3682 f3683 f3684 f3685 f3686 f3687 f3688 f3689 f369 f3690 f3691 f3692 f3693 f3694 f3695 f3696 f3697 f3698 f3699 f37 f370 f3700 f3701 f3702 f3703 f3704 f3705 f3706 f3707 f3708 f3709 f371 f3710 f3711 f3712 f3713 f3714 f3715 f3716 f3717 f3718 f3719 f372 f3720 f3721 f3722 f3723 f3724 f3725 f3726 f3727 f3728 f3729 f373 f3730 f3731 f3732 f3733 f3734 f3735 f3736 f3737 f3738 f3739 f374 f3740 f3741 f3742 f3743 f3744 f3745 f3746 f3747 f3748 f3749 f375 f3750 f3751 f3752 f3753 f3754 f3755 f3756 f3757 f3758 f3759 f376 f3760 f3761 f3762 f3763 f3764 f3765 f3766 f3767 f3768 f3769 f377 f3770 f3771 f3772 f3773 f3774 f3775 f3776 f3777 f3778 f3779 f378 f3780 f3781 f3782 f3783 f3784 f3785 f3786 f3787 f3788 f3789 f379 f3790 f3791 f3792 f3793 f3794 f3795 f3796 f3797 f3798 f3799 f38 f380 f3800 f3801 f3802 f3803 f3804 f3805 f3806 f3807 f3808 f3809 f381 f3810 f3811 f3812 f3813 f3814 f3815 f3816 f3817 f3818 f3819 f382 f3820 f3821 f3822 f3823 f3824 f3825 f3826 f3827 f3828 f3829 f383 f3830 f3831 f3832 f3833 f3834 f3835 f3836 f3837 f3838 f3839 f384 f3840 f3841 f3842 f3843 f3844 f3845 f3846 f3847 f3848 f3849 f385 f3850 f3851 f3852 f3853 f3854 f3855 f3856 f3857 f3858 f3859 f386 f3860 f3861 f3862 f3863 f3864 f3865 f3866 f3867 f3868 f3869 f387 f3870 f3871 f3872 f3873 f3874 f3875 f3876 f3877 f3878 f3879 f388 f3880 f3881 f3882 f3883 f3884 f3885 f3886 f3887 f3888 f3889 f389 f3890 f3891 f3892 f3893 f3894 f3895 f3896 f3897 f3898 f3899 f39 f390 f3900 f3901 f3902 f3903 f3904 f3905 f3906 f3907 f3908 f3909 f391 f3910 f3911 f3912 f3913 f3914 f3915 f3916 f3917 f3918 f3919 f392 f3920 f3921 f3922 f3923 f3924 f3925 f3926 f3927 f3928 f3929 f393 f3930 f3931 f3932 f3933 f3934 f3935 f3936 f3937 f3938 f3939 f394 f3940 f3941 f3942 f3943 f3944 f3945 f3946 f3947 f3948 f3949 f395 f3950 f3951 f3952 f3953 f3954 f3955 f3956 f3957 f3958 f3959 f396 f3960 f3961 f3962 f3963 f3964 f3965 f3966 f3967 f3968 f3969 f397 f3970 f3971 f3972 f3973 f3974 f3975 f3976 f3977 f3978 f3979 f398 f3980 f3981 f3982 f3983 f3984 f3985 f3986 f3987 f3988 f3989 f399 f3990 f3991 f3992 f3993 f3994 f3995 f3996 f3997 f3998 f3999 f4 f40 f400 f4000 f4001 f4002 f4003 f4004 f4005 f4006 f4007 f4008 f4009 f401 f4010 f4011 f4012 f4013 f4014 f4015 f4016 f4017 f4018 f4019 f402 f4020 f4021 f4022 f4023 f4024 f4025 f4026 f4027 f4028 f4029 f403 f4030 f4031 f4032 f4033 f4034 f4035 f4036 f4037 f4038 f4039 f404 f4040 f4041 f4042 f4043 f4044 f4045 f4046 f4047 f4048 f4049 f405 f4050 f4051 f4052 f4053 f4054 f4055 f4056 f4057 f4058 f4059 f406 f4060 f4061 f4062 f4063 f4064 f4065 f4066 f4067 f4068 f4069 f407 f4070 f4071 f4072 f4073 f4074 f4075 f4076 f4077 f4078 f4079 f408 f4080 f4081 f4082 f4083 f4084 f4085 f4086 f4087 f4088 f4089 f409 f4090 f4091 f4092 f4093 f4094 f4095 f4096 f4097 f4098 f4099 f41 f410 f4100 f4101 f4102 f4103 f4104 f4105 f4106 f4107 f4108 f4109 f411 f4110 f4111 f4112 f4113 f4114 f4115 f4116 f4117 f4118 f4119 f412 f4120 f4121 f4122 f4123 f4124 f4125 f4126 f4127 f4128 f4129 f413 f4130 f4131 f4132 f4133 f4134 f4135 f4136 f4137 f4138 f4139 f414 f4140 f4141 f4142 f4143 f4144 f4145 f4146 f4147 f4148 f4149 f415 f4150 f4151 f4152 f4153 f4154 f4155 f4156 f4157 f4158 f4159 f416 f4160 f4161 f4162 f4163 f4164 f4165 f4166 f4167 f4168 f4169 f417 f4170 f4171 f4172 f4173 f4174 f4175 f4176 f4177 f4178 f4179 f418 f4180 f4181 f4182 f4183 f4184 f4185 f4186 f4187 f4188 f4189 f419 f4190 f4191 f4192 f4193 f4194 f4195 f4196 f4197 f4198 f4199 f42 f420 f4200 f4201 f4202 f4203 f4204 f4205 f4206 f4207 f4208 f4209 f421 f4210 f4211 f4212 f4213 f4214 f4215 f4216 f4217 f4218 f4219 f422 f4220 f4221 f4222 f4223 f4224 f4225 f4226 f4227 f4228 f4229 f423 f4230 f4231 f4232 f4233 f4234 f4235 f4236 f4237 f4238 f4239 f424 f4240 f4241 f4242 f4243 f4244 f4245 f4246 f4247 f4248 f4249 f425 f4250 f4251 f4252 f4253 f4254 f4255 f4256 f4257 f4258 f4259 f426 f4260 f4261 f4262 f4263 f4264 f4265 f4266 f4267 f4268 f4269 f427 f4270 f4271 f4272 f4273 f4274 f4275 f4276 f4277 f4278 f4279 f428 f4280 f4281 f4282 f4283 f4284 f4285 f4286 f4287 f4288 f4289 f429 f4290 f4291 f4292 f4293 f4294 f4295 f4296 f4297 f4298 f4299 f43 f430 f4300 f4301 f4302 f4303 f4304 f4305 f4306 f4307 f4308 f4309 f431 f4310 f4311 f4312 f4313 f4314 f4315 f4316 f4317 f4318 f4319 f432 f4320 f4321 f4322 f4323 f4324 f4325 f4326 f4327 f4328 f4329 f433 f4330 f4331 f4332 f4333 f4334 f4335 f4336 f4337 f4338 f4339 f434 f4340 f4341 f4342 f4343 f4344 f4345 f4346 f4347 f4348 f4349 f435 f4350 f4351 f4352 f4353 f4354 f4355 f4356 f4357 f4358 f4359 f436 f4360 f4361 f4362 f4363 f4364 f4365 f4366 f4367 f4368 f4369 f437 f4370 f4371 f4372 f4373 f4374 f4375 f4376 f4377 f4378 f4379 f438 f4380 f4381 f4382 f4383 f4384 f4385 f4386 f4387 f4388 f4389 f439 f4390 f4391 f4392 f4393 f4394 f4395 f4396 f4397 f4398 f4399 f44 f440 f4400 f4401 f4402 f4403 f4404 f4405 f4406 f4407 f4408 f4409 f441 f4410 f4411 f4412 f4413 f4414 f4415 f4416 f4417 f4418 f4419 f442 f4420 f4421 f4422 f4423 f4424 f4425 f4426 f4427 f4428 f4429 f443 f4430 f4431 f4432 f4433 f4434 f4435 f4436 f4437 f4438 f4439 f444 f4440 f4441 f4442 f4443 f4444 f4445 f4446 f4447 f4448 f4449 f445 f4450 f4451 f4452 f4453 f4454 f4455 f4456 f4457 f4458 f4459 f446 f4460 f4461 f4462 f4463 f4464 f4465 f4466 f4467 f4468 f4469 f447 f4470 f4471 f4472 f4473 f4474 f4475 f4476 f4477 f4478 f4479 f448 f4480 f4481 f4482 f4483 f4484 f4485 f4486 f4487 f4488 f4489 f449 f4490 f4491 f4492 f4493 f4494 f4495 f4496 f4497 f4498 f4499 f45 f450 f4500 f4501 f4502 f4503 f4504 f4505 f4506 f4507 f4508 f4509 f451 f4510 f4511 f4512 f4513 f4514 f4515 f4516 f4517 f4518 f4519 f452 f4520 f4521 f4522 f4523 f4524 f4525 f4526 f4527 f4528 f4529 f453 f4530 f4531 f4532 f4533 f4534 f4535 f4536 f4537 f4538 f4539 f454 f4540 f4541 f4542 f4543 f4544 f4545 f4546 f4547 f4548 f4549 f455 f4550 f4551 f4552 f4553 f4554 f4555 f4556 f4557 f4558 f4559 f456 f4560 f4561 f4562 f4563 f4564 f4565 f4566 f4567 f4568 f4569 f457 f4570 f4571 f4572 f4573 f4574 f4575 f4576 f4577 f4578 f4579 f458 f4580 f4581 f4582 f4583 f4584 f4585 f4586 f4587 f4588 f4589 f459 f4590 f4591 f4592 f4593 f4594 f4595 f4596 f4597 f4598 f4599 f46 f460 f4600 f4601 f4602 f4603 f4604 f4605 f4606 f4607 f4608 f4609 f461 f4610 f4611 f4612 f4613 f4614 f4615 f4616 f4617 f4618 f4619 f462 f4620 f4621 f4622 f4623 f4624 f4625 f4626 f4627 f4628 f4629 f463 f4630 f4631 f4632 f4633 f4634 f4635 f4636 f4637 f4638 f4639 f464 f4640 f4641 f4642 f4643 f4644 f4645 f4646 f4647 f4648 f4649 f465 f4650 f4651 f4652 f4653 f4654 f4655 f4656 f4657 f4658 f4659 f466 f4660 f4661 f4662 f4663 f4664 f4665 f4666 f4667 f4668 f4669 f467 f4670 f4671 f4672 f4673 f4674 f4675 f4676 f4677 f4678 f4679 f468 f4680 f4681 f4682 f4683 f4684 f4685 f4686 f4687 f4688 f4689 f469 f4690 f4691 f4692 f4693 f4694 f4695 f4696 f4697 f4698 f4699 f47 f470 f4700 f4701 f4702 f4703 f4704 f4705 f4706 f4707 f4708 f4709 f471 f4710 f4711 f4712 f4713 f4714 f4715 f4716 f4717 f4718 f4719 f472 f4720 f4721 f4722 f4723 f4724 f4725 f4726 f4727 f4728 f4729 f473 f4730 f4731 f4732 f4733 f4734 f4735 f4736 f4737 f4738 f4739 f474 f4740 f4741 f4742 f4743 f4744 f4745 f4746 f4747 f4748 f4749 f475 f4750 f4751 f4752 f4753 f4754 f4755 f4756 f4757 f4758 f4759 f476 f4760 f4761 f4762 f4763 f4764 f4765 f4766 f4767 f4768 f4769 f477 f4770 f4771 f4772 f4773 f4774 f4775 f4776 f4777 f4778 f4779 f478 f4780 f4781 f4782 f4783 f4784 f4785 f4786 f4787 f4788 f4789 f479 f4790 f4791 f4792 f4793 f4794 f4795 f4796 f4797 f4798 f4799 f48 f480 f4800 f4801 f4802 f4803 f4804 f4805 f4806 f4807 f4808 f4809 f481 f4810 f4811 f4812 f4813 f4814 f4815 f4816 f4817 f4818 f4819 f482 f4820 f4821 f4822 f4823 f4824 f4825 f4826 f4827 f4828 f4829 f483 f4830 f4831 f4832 f4833 f4834 f4835 f4836 f4837 f4838 f4839 f484 f4840 f4841 f4842 f4843 f4844 f4845 f4846 f4847 f4848 f4849 f485 f4850 f4851 f4852 f4853 f4854 f4855 f4856 f4857 f4858 f4859 f486 f4860 f4861 f4862 f4863 f4864 f4865 f4866 f4867 f4868 f4869 f487 f4870 f4871 f4872 f4873 f4874 f4875 f4876 f4877 f4878 f4879 f488 f4880 f4881 f4882 f4883 f4884 f4885 f4886 f4887 f4888 f4889 f489 f4890 f4891 f4892 f4893 f4894 f4895 f4896 f4897 f4898 f4899 f49 f490 f4900 f4901 f4902 f4903 f4904 f4905 f4906 f4907 f4908 f4909 f491 f4910 f4911 f4912 f4913 f4914 f4915 f4916 f4917 f4918 f4919 f492 f4920 f4921 f4922 f4923 f4924 f4925 f4926 f4927 f4928 f4929 f493 f4930 f4931 f4932 f4933 f4934 f4935 f4936 f4937 f4938 f4939 f494 f4940 f4941 f4942 f4943 f4944 f4945 f4946 f4947 f4948 f4949 f495 f4950 f4951 f4952 f4953 f4954 f4955 f4956 f4957 f4958 f4959 f496 f4960 f4961 f4962 f4963 f4964 f4965 f4966 f4967 f4968 f4969 f497 f4970 f4971 f4972 f4973 f4974 f4975 f4976 f4977 f4978 f4979 f498 f4980 f4981 f4982 f4983 f4984 f4985 f4986 f4987 f4988 f4989 f499 f4990 f4991 f4992 f4993 f4994 f4995 f4996 f4997 f4998 f4999 f5 f50 f500 f5000 f5001 f5002 f5003 f5004 f5005 f5006 f5007 f5008 f5009 f501 f5010 f5011 f5012 f5013 f5014 f5015 f5016 f5017 f5018 f5019 f502 f5020 f5021 f5022 f5023 f5024 f5025 f5026 f5027 f5028 f5029 f503 f5030 f5031 f5032 f5033 f5034 f5035 f5036 f5037 f5038 f5039 f504 f5040 f5041 f5042 f5043 f5044 f5045 f5046 f5047 f5048 f5049 f505 f5050 f5051 f5052 f5053 f5054 f5055 f5056 f5057 f5058 f5059 f506 f5060 f5061 f5062 f5063 f5064 f5065 f5066 f5067 f5068 f5069 f507 f5070 f5071 f5072 f5073 f5074 f5075 f5076 f5077 f5078 f5079 f508 f5080 f5081 f5082 f5083 f5084 f5085 f5086 f5087 f5088 f5089 f509 f5090 f5091 f5092 f5093 f5094 f5095 f5096 f5097 f5098 f5099 f51 f510 f5100 f5101 f5102 f5103 f5104 f5105 f5106 f5107 f5108 f5109 f511 f5110 f5111 f5112 f5113 f5114 f5115 f5116 f5117 f5118 f5119 f512 f5120 f5121 f5122 f5123 f5124 f5125 f5126 f5127 f5128 f5129 f513 f5130 f5131 f5132 f5133 f5134 f5135 f5136 f5137 f5138 f5139 f514 f5140 f5141 f5142 f5143 f5144 f5145 f5146 f5147 f5148 f5149 f515 f5150 f5151 f5152 f5153 f5154 f5155 f5156 f5157 f5158 f5159 f516 f5160 f5161 f5162 f5163 f5164 f5165 f5166 f5167 f5168 f5169 f517 f5170 f5171 f5172 f5173 f5174 f5175 f5176 f5177 f5178 f5179 f518 f5180 f5181 f5182 f5183 f5184 f5185 f5186 f5187 f5188 f5189 f519 f5190 f5191 f5192 f5193 f5194 f5195 f5196 f5197 f5198 f5199 f52 f520 f5200 f5201 f5202 f5203 f5204 f5205 f5206 f5207 f5208 f5209 f521 f5210 f5211 f5212 f5213 f5214 f5215 f5216 f5217 f5218 f5219 f522 f5220 f5221 f5222 f5223 f5224 f5225 f5226 f5227 f5228 f5229 f523 f5230 f5231 f5232 f5233 f5234 f5235 f5236 f5237 f5238 f5239 f524 f5240 f5241 f5242 f5243 f5244 f5245 f5246 f5247 f5248 f5249 f525 f5250 f5251 f5252 f5253 f5254 f5255 f5256 f5257 f5258 f5259 f526 f5260 f5261 f5262 f5263 f5264 f5265 f5266 f5267 f5268 f5269 f527 f5270 f5271 f5272 f5273 f5274 f5275 f5276 f5277 f5278 f5279 f528 f5280 f5281 f5282 f5283 f5284 f5285 f5286 f5287 f5288 f5289 f529 f5290 f5291 f5292 f5293 f5294 f5295 f5296 f5297 f5298 f5299 f53 f530 f5300 f5301 f5302 f5303 f5304 f5305 f5306 f5307 f5308 f5309 f531 f5310 f5311 f5312 f5313 f5314 f5315 f5316 f5317 f5318 f5319 f532 f5320 f5321 f5322 f5323 f5324 f5325 f5326 f5327 f5328 f5329 f533 f5330 f5331 f5332 f5333 f5334 f5335 f5336 f5337 f5338 f5339 f534 f5340 f5341 f5342 f5343 f5344 f5345 f5346 f5347 f5348 f5349 f535 f5350 f5351 f5352 f5353 f5354 f5355 f5356 f5357 f5358 f5359 f536 f5360 f5361 f5362 f5363 f5364 f5365 f5366 f5367 f5368 f5369 f537 f5370 f5371 f5372 f5373 f5374 f5375 f5376 f5377 f5378 f5379 f538 f5380 f5381 f5382 f5383 f5384 f5385 f5386 f5387 f5388 f5389 f539 f5390 f5391 f5392 f5393 f5394 f5395 f5396 f5397 f5398 f5399 f54 f540 f5400 f5401 f5402 f5403 f5404 f5405 f5406 f5407 f5408 f5409 f541 f5410 f5411 f5412 f5413 f5414 f5415 f5416 f5417 f5418 f5419 f542 f5420 f5421 f5422 f5423 f5424 f5425 f5426 f5427 f5428 f5429 f543 f5430 f5431 f5432 f5433 f5434 f5435 f5436 f5437 f5438 f5439 f544 f5440 f5441 f5442 f5443 f5444 f5445 f5446 f5447 f5448 f5449 f545 f5450 f5451 f5452 f5453 f5454 f5455 f5456 f5457 f5458 f5459 f546 f5460 f5461 f5462 f5463 f5464 f5465 f5466 f5467 f5468 f5469 f547 f5470 f5471 f5472 f5473 f5474 f5475 f5476 f5477 f5478 f5479 f548 f5480 f5481 f5482 f5483 f5484 f5485 f5486 f5487 f5488 f5489 f549 f5490 f5491 f5492 f5493 f5494 f5495 f5496 f5497 f5498 f5499 f55 f550 f5500 f5501 f5502 f5503 f5504 f5505 f5506 f5507 f5508 f5509 f551 f5510 f5511 f5512 f5513 f5514 f5515 f5516 f5517 f5518 f5519 f552 f5520 f5521 f5522 f5523 f5524 f5525 f5526 f5527 f5528 f5529 f553 f5530 f5531 f5532 f5533 f5534 f5535 f5536 f5537 f5538 f5539 f554 f5540 f5541 f5542 f5543 f5544 f5545 f5546 f5547 f5548 f5549 f555 f5550 f5551 f5552 f5553 f5554 f5555 f5556 f5557 f5558 f5559 f556 f5560 f5561 f5562 f5563 f5564 f5565 f5566 f5567 f5568 f5569 f557 f5570 f5571 f5572 f5573 f5574 f5575 f5576 f5577 f5578 f5579 f558 f5580 f5581 f5582 f5583 f5584 f5585 f5586 f5587 f5588 f5589 f559 f5590 f5591 f5592 f5593 f5594 f5595 f5596 f5597 f5598 f5599 f56 f560 f5600 f5601 f5602 f5603 f5604 f5605 f5606 f5607 f5608 f5609 f561 f5610 f5611 f5612 f5613 f5614 f5615 f5616 f5617 f5618 f5619 f562 f5620 f5621 f5622 f5623 f5624 f5625 f5626 f5627 f5628 f5629 f563 f5630 f5631 f5632 f5633 f5634 f5635 f5636 f5637 f5638 f5639 f564 f5640 f5641 f5642 f5643 f5644 f5645 f5646 f5647 f5648 f5649 f565 f5650 f5651 f5652 f5653 f5654 f5655 f5656 f5657 f5658 f5659 f566 f5660 f5661 f5662 f5663 f5664 f5665 f5666 f5667 f5668 f5669 f567 f5670 f5671 f5672 f5673 f5674 f5675 f5676 f5677 f5678 f5679 f568 f5680 f5681 f5682 f5683 f5684 f5685 f5686 f5687 f5688 f5689 f569 f5690 f5691 f5692 f5693 f5694 f5695 f5696 f5697 f5698 f5699 f57 f570 f5700 f5701 f5702 f5703 f5704 f5705 f5706 f5707 f5708 f5709 f571 f5710 f5711 f5712 f5713 f5714 f5715 f5716 f5717 f5718 f5719 f572 f5720 f5721 f5722 f5723 f5724 f5725 f5726 f5727 f5728 f5729 f573 f5730 f5731 f5732 f5733 f5734 f5735 f5736 f5737 f5738 f5739 f574 f5740 f5741 f5742 f5743 f5744 f5745 f5746 f5747 f5748 f5749 f575 f5750 f5751 f5752 f5753 f5754 f5755 f5756 f5757 f5758 f5759 f576 f5760 f5761 f5762 f5763 f5764 f5765 f5766 f5767 f5768 f5769 f577 f5770 f5771 f5772 f5773 f5774 f5775 f5776 f5777 f5778 f5779 f578 f5780 f5781 f5782 f5783 f5784 f5785 f5786 f5787 f5788 f5789 f579 f5790 f5791 f5792 f5793 f5794 f5795 f5796 f5797 f5798 f5799 f58 f580 f5800 f5801 f5802 f5803 f5804 f5805 f5806 f5807 f5808 f5809 f581 f5810 f5811 f5812 f5813 f5814 f5815 f5816 f5817 f5818 f5819 f582 f5820 f5821 f5822 f5823 f5824 f5825 f5826 f5827 f5828 f5829 f583 f5830 f5831 f5832 f5833 f5834 f5835 f5836 f5837 f5838 f5839 f584 f5840 f5841 f5842 f5843 f5844 f5845 f5846 f5847 f5848 f5849 f585 f5850 f5851 f5852 f5853 f5854 f5855 f5856 f5857 f5858 f5859 f586 f5860 f5861 f5862 f5863 f5864 f5865 f5866 f5867 f5868 f5869 f587 f5870 f5871 f5872 f5873 f5874 f5875 f5876 f5877 f5878 f5879 f588 f5880 f5881 f5882 f5883 f5884 f5885 f5886 f5887 f5888 f5889 f589 f5890 f5891 f5892 f5893 f5894 f5895 f5896 f5897 f5898 f5899 f59 f590 f5900 f5901 f5902 f5903 f5904 f5905 f5906 f5907 f5908 f5909 f591 f5910 f5911 f5912 f5913 f5914 f5915 f5916 f5917 f5918 f5919 f592 f5920 f5921 f5922 f5923 f5924 f5925 f5926 f5927 f5928 f5929 f593 f5930 f5931 f5932 f5933 f5934 f5935 f5936 f5937 f5938 f5939 f594 f5940 f5941 f5942 f5943 f5944 f5945 f5946 f5947 f5948 f5949 f595 f5950 f5951 f5952 f5953 f5954 f5955 f5956 f5957 f5958 f5959 f596 f5960 f5961 f5962 f5963 f5964 f5965 f5966 f5967 f5968 f5969 f597 f5970 f5971 f5972 f5973 f5974 f5975 f5976 f5977 f5978 f5979 f598 f5980 f5981 f5982 f5983 f5984 f5985 f5986 f5987 f5988 f5989 f599 f5990 f5991 f5992 f5993 f5994 f5995 f5996 f5997 f5998 f5999 f6 f60 f600 f6000 f6001 f6002 f6003 f6004 f6005 f6006 f6007 f6008 f6009 f601 f6010 f6011 f6012 f6013 f6014 f6015 f6016 f6017 f6018 f6019 f602 f6020 f6021 f6022 f6023 f6024 f6025 f6026 f6027 f6028 f6029 f603 f6030 f6031 f6032 f6033 f6034 f6035 f6036 f6037 f6038 f6039 f604 f6040 f6041 f6042 f6043 f6044 f6045 f6046 f6047 f6048 f6049 f605 f6050 f6051 f6052 f6053 f6054 f6055 f6056 f6057 f6058 f6059 f606 f6060 f6061 f6062 f6063 f6064 f6065 f6066 f6067 f6068 f6069 f607 f6070 f6071 f6072 f6073 f6074 f6075 f6076 f6077 f6078 f6079 f608 f6080 f6081 f6082 f6083 f6084 f6085 f6086 f6087 f6088 f6089 f609 f6090 f6091 f6092 f6093 f6094 f6095 f6096 f6097 f6098 f6099 f61 f610 f6100 f6101 f6102 f6103 f6104 f6105 f6106 f6107 f6108 f6109 f611 f6110 f6111 f6112 f6113 f6114 f6115 f6116 f6117 f6118 f6119 f612 f6120 f6121 f6122 f6123 f6124 f6125 f6126 f6127 f6128 f6129 f613 f6130 f6131 f6132 f6133 f6134 f6135 f6136 f6137 f6138 f6139 f614 f6140 f6141 f6142 f6143 f6144 f6145 f6146 f6147 f6148 f6149 f615 f6150 f6151 f6152 f6153 f6154 f6155 f6156 f6157 f6158 f6159 f616 f6160 f6161 f6162 f6163 f6164 f6165 f6166 f6167 f6168 f6169 f617 f6170 f6171 f6172 f6173 f6174 f6175 f6176 f6177 f6178 f6179 f618 f6180 f6181 f6182 f6183 f6184 f6185 f6186 f6187 f6188 f6189 f619 f6190 f6191 f6192 f6193 f6194 f6195 f6196 f6197 f6198 f6199 f62 f620 f6200 f6201 f6202 f6203 f6204 f6205 f6206 f6207 f6208 f6209 f621 f6210 f6211 f6212 f6213 f6214 f6215 f6216 f6217 f6218 f6219 f622 f6220 f6221 f6222 f6223 f6224 f6225 f6226 f6227 f6228 f6229 f623 f6230 f6231 f6232 f6233 f6234 f6235 f6236 f6237 f6238 f6239 f624 f6240 f6241 f6242 f6243 f6244 f6245 f6246 f6247 f6248 f6249 f625 f6250 f6251 f6252 f6253 f6254 f6255 f6256 f6257 f6258 f6259 f626 f6260 f6261 f6262 f6263 f6264 f6265 f6266 f6267 f6268 f6269 f627 f6270 f6271 f6272 f6273 f6274 f6275 f6276 f6277 f6278 f6279 f628 f6280 f6281 f6282 f6283 f6284 f6285 f6286 f6287 f6288 f6289 f629 f6290 f6291 f6292 f6293 f6294 f6295 f6296 f6297 f6298 f6299 f63 f630 f6300 f6301 f6302 f6303 f6304 f6305 f6306 f6307 f6308 f6309 f631 f6310 f6311 f6312 f6313 f6314 f6315 f6316 f6317 f6318 f6319 f632 f6320 f6321 f6322 f6323 f6324 f6325 f6326 f6327 f6328 f6329 f633 f6330 f6331 f6332 f6333 f6334 f6335 f6336 f6337 f6338 f6339 f634 f6340 f6341 f6342 f6343 f6344 f6345 f6346 f6347 f6348 f6349 f635 f6350 f6351 f6352 f6353 f6354 f6355 f6356 f6357 f6358 f6359 f636 f6360 f6361 f6362 f6363 f6364 f6365 f6366 f6367 f6368 f6369 f637 f6370 f6371 f6372 f6373 f6374 f6375 f6376 f6377 f6378 f6379 f638 f6380 f6381 f6382 f6383 f6384 f6385 f6386 f6387 f6388 f6389 f639 f6390 f6391 f6392 f6393 f6394 f6395 f6396 f6397 f6398 f6399 f64 f640 f6400 f6401 f6402 f6403 f6404 f6405 f6406 f6407 f6408 f6409 f641 f6410 f6411 f6412 f6413 f6414 f6415 f6416 f6417 f6418 f6419 f642 f6420 f6421 f6422 f6423 f6424 f6425 f6426 f6427 f6428 f6429 f643 f6430 f6431 f6432 f6433 f6434 f6435 f6436 f6437 f6438 f6439 f644 f6440 f6441 f6442 f6443 f6444 f6445 f6446 f6447 f6448 f6449 f645 f6450 f6451 f6452 f6453 f6454 f6455 f6456 f6457 f6458 f6459 f646 f6460 f6461 f6462 f6463 f6464 f6465 f6466 f6467 f6468 f6469 f647 f6470 f6471 f6472 f6473 f6474 f6475 f6476 f6477 f6478 f6479 f648 f6480 f6481 f6482 f6483 f6484 f6485 f6486 f6487 f6488 f6489 f649 f6490 f6491 f6492 f6493 f6494 f6495 f6496 f6497 f6498 f6499 f65 f650 f6500 f6501 f6502 f6503 f6504 f6505 f6506 f6507 f6508 f6509 f651 f6510 f6511 f6512 f6513 f6514 f6515 f6516 f6517 f6518 f6519 f652 f6520 f6521 f6522 f6523 f6524 f6525 f6526 f6527 f6528 f6529 f653 f6530 f6531 f6532 f6533 f6534 f6535 f6536 f6537 f6538 f6539 f654 f6540 f6541 f6542 f6543 f6544 f6545 f6546 f6547 f6548 f6549 f655 f6550 f6551 f6552 f6553 f6554 f6555 f6556 f6557 f6558 f6559 f656 f6560 f6561 f6562 f6563 f6564 f6565 f6566 f6567 f6568 f6569 f657 f6570 f6571 f6572 f6573 f6574 f6575 f6576 f6577 f6578 f6579 f658 f6580 f6581 f6582 f6583 f6584 f6585 f6586 f6587 f6588 f6589 f659 f6590 f6591 f6592 f6593 f6594 f6595 f6596 f6597 f6598 f6599 f66 f660 f6600 f6601 f6602 f6603 f6604 f6605 f6606 f6607 f6608 f6609 f661 f6610 f6611 f6612 f6613 f6614 f6615 f6616 f6617 f6618 f6619 f662 f6620 f6621 f6622 f6623 f6624 f6625 f6626 f6627 f6628 f6629 f663 f6630 f6631 f6632 f6633 f6634 f6635 f6636 f6637 f6638 f6639 f664 f6640 f6641 f6642 f6643 f6644 f6645 f6646 f6647 f6648 f6649 f665 f6650 f6651 f6652 f6653 f6654 f6655 f6656 f6657 f6658 f6659 f666 f6660 f6661 f6662 f6663 f6664 f6665 f6666 f6667 f6668 f6669 f667 f6670 f6671 f6672 f6673 f6674 f6675 f6676 f6677 f6678 f6679 f668 f6680 f6681 f6682 f6683 f6684 f6685 f6686 f6687 f6688 f6689 f669 f6690 f6691 f6692 f6693 f6694 f6695 f6696 f6697 f6698 f6699 f67 f670 f6700 f6701 f6702 f6703 f6704 f6705 f6706 f6707 f6708 f6709 f671 f6710 f6711 f6712 f6713 f6714 f6715 f6716 f6717 f6718 f6719 f672 f6720 f6721 f6722 f6723 f6724 f6725 f6726 f6727 f6728 f6729 f673 f6730 f6731 f6732 f6733 f6734 f6735 f6736 f6737 f6738 f6739 f674 f6740 f6741 f6742 f6743 f6744 f6745 f6746 f6747 f6748 f6749 f675 f6750 f6751 f6752 f6753 f6754 f6755 f6756 f6757 f6758 f6759 f676 f6760 f6761 f6762 f6763 f6764 f6765 f6766 f6767 f6768 f6769 f677 f6770 f6771 f6772 f6773 f6774 f6775 f6776 f6777 f6778 f6779 f678 f6780 f6781 f6782 f6783 f6784 f6785 f6786 f6787 f6788 f6789 f679 f6790 f6791 f6792 f6793 f6794 f6795 f6796 f6797 f6798 f6799 f68 f680 f6800 f6801 f6802 f6803 f6804 f6805 f6806 f6807 f6808 f6809 f681 f6810 f6811 f6812 f6813 f6814 f6815 f6816 f6817 f6818 f6819 f682 f6820 f6821 f6822 f6823 f6824 f6825 f6826 f6827 f6828 f6829 f683 f6830 f6831 f6832 f6833 f6834 f6835 f6836 f6837 f6838 f6839 f684 f6840 f6841 f6842 f6843 f6844 f6845 f6846 f6847 f6848 f6849 f685 f6850 f6851 f6852 f6853 f6854 f6855 f6856 f6857 f6858 f6859 f686 f6860 f6861 f6862 f6863 f6864 f6865 f6866 f6867 f6868 f6869 f687 f6870 f6871 f6872 f6873 f6874 f6875 f6876 f6877 f6878 f6879 f688 f6880 f6881 f6882 f6883 f6884 f6885 f6886 f6887 f6888 f6889 f689 f6890 f6891 f6892 f6893 f6894 f6895 f6896 f6897 f6898 f6899 f69 f690 f6900 f6901 f6902 f6903 f6904 f6905 f6906 f6907 f6908 f6909 f691 f6910 f6911 f6912 f6913 f6914 f6915 f6916 f6917 f6918 f6919 f692 f6920 f6921 f6922 f6923 f6924 f6925 f6926 f6927 f6928 f6929 f693 f6930 f6931 f6932 f6933 f6934 f6935 f6936 f6937 f6938 f6939 f694 f6940 f6941 f6942 f6943 f6944 f6945 f6946 f6947 f6948 f6949 f695 f6950 f6951 f6952 f6953 f6954 f6955 f6956 f6957 f6958 f6959 f696 f6960 f6961 f6962 f6963 f6964 f6965 f6966 f6967 f6968 f6969 f697 f6970 f6971 f6972 f6973 f6974 f6975 f6976 f6977 f6978 f6979 f698 f6980 f6981 f6982 f6983 f6984 f6985 f6986 f6987 f6988 f6989 f699 f6990 f6991 f6992 f6993 f6994 f6995 f6996 f6997 f6998 f6999 f7 f70 f700 f7000 f7001 f7002 f7003 f7004 f7005 f7006 f7007 f7008 f7009 f701 f7010 f7011 f7012 f7013 f7014 f7015 f7016 f7017 f7018 f7019 f702 f7020 f7021 f7022 f7023 f7024 f7025 f7026 f7027 f7028 f7029 f703 f7030 f7031 f7032 f7033 f7034 f7035 f7036 f7037 f7038 f7039 f704 f7040 f7041 f7042 f7043 f7044 f7045 f7046 f7047 f7048 f7049 f705 f7050 f7051 f7052 f7053 f7054 f7055 f7056 f7057 f7058 f7059 f706 f7060 f7061 f7062 f7063 f7064 f7065 f7066 f7067 f7068 f7069 f707 f7070 f7071 f7072 f7073 f7074 f7075 f7076 f7077 f7078 f7079 f708 f7080 f7081 f7082 f7083 f7084 f7085 f7086 f7087 f7088 f7089 f709 f7090 f7091 f7092 f7093 f7094 f7095 f7096 f7097 f7098 f7099 f71 f710 f7100 f7101 f7102 f7103 f7104 f7105 f7106 f7107 f7108 f7109 f711 f7110 f7111 f7112 f7113 f7114 f7115 f7116 f7117 f7118 f7119 f712 f7120 f7121 f7122 f7123 f7124 f7125 f7126 f7127 f7128 f7129 f713 f7130 f7131 f7132 f7133 f7134 f7135 f7136 f7137 f7138 f7139 f714 f7140 f7141 f7142 f7143 f7144 f7145 f7146 f7147 f7148 f7149 f715 f7150 f7151 f7152 f7153 f7154 f7155 f7156 f7157 f7158 f7159 f716 f7160 f7161 f7162 f7163 f7164 f7165 f7166 f7167 f7168 f7169 f717 f7170 f7171 f7172 f7173 f7174 f7175 f7176 f7177 f7178 f7179 f718 f7180 f7181 f7182 f7183 f7184 f7185 f7186 f7187 f7188 f7189 f719 f7190 f7191 f7192 f7193 f7194 f7195 f7196 f7197 f7198 f7199 f72 f720 f7200 f7201 f7202 f7203 f7204 f7205 f7206 f7207 f7208 f7209 f721 f7210 f7211 f7212 f7213 f7214 f7215 f7216 f7217 f7218 f7219 f722 f7220 f7221 f7222 f7223 f7224 f7225 f7226 f7227 f7228 f7229 f723 f7230 f7231 f7232 f7233 f7234 f7235 f7236 f7237 f7238 f7239 f724 f7240 f7241 f7242 f7243 f7244 f7245 f7246 f7247 f7248 f7249 f725 f7250 f7251 f7252 f7253 f7254 f7255 f7256 f7257 f7258 f7259 f726 f7260 f7261 f7262 f7263 f7264 f7265 f7266 f7267 f7268 f7269 f727 f7270 f7271 f7272 f7273 f7274 f7275 f7276 f7277 f7278 f7279 f728 f7280 f7281 f7282 f7283 f7284 f7285 f7286 f7287 f7288 f7289 f729 f7290 f7291 f7292 f7293 f7294 f7295 f7296 f7297 f7298 f7299 f73 f730 f7300 f7301 f7302 f7303 f7304 f7305 f7306 f7307 f7308 f7309 f731 f7310 f7311 f7312 f7313 f7314 f7315 f7316 f7317 f7318 f7319 f732 f7320 f7321 f7322 f7323 f7324 f7325 f7326 f7327 f7328 f7329 f733 f7330 f7331 f7332 f7333 f7334 f7335 f7336 f7337 f7338 f7339 f734 f7340 f7341 f7342 f7343 f7344 f7345 f7346 f7347 f7348 f7349 f735 f7350 f7351 f7352 f7353 f7354 f7355 f7356 f7357 f7358 f7359 f736 f7360 f7361 f7362 f7363 f7364 f7365 f7366 f7367 f7368 f7369 f737 f7370 f7371 f7372 f7373 f7374 f7375 f7376 f7377 f7378 f7379 f738 f7380 f7381 f7382 f7383 f7384 f7385 f7386 f7387 f7388 f7389 f739 f7390 f7391 f7392 f7393 f7394 f7395 f7396 f7397 f7398 f7399 f74 f740 f7400 f7401 f7402 f7403 f7404 f7405 f7406 f7407 f7408 f7409 f741 f7410 f7411 f7412 f7413 f7414 f7415 f7416 f7417 f7418 f7419 f742 f7420 f7421 f7422 f7423 f7424 f7425 f7426 f7427 f7428 f7429 f743 f7430 f7431 f7432 f7433 f7434 f7435 f7436 f7437 f7438 f7439 f744 f7440 f7441 f7442 f7443 f7444 f7445 f7446 f7447 f7448 f7449 f745 f7450 f7451 f7452 f7453 f7454 f7455 f7456 f7457 f7458 f7459 f746 f7460 f7461 f7462 f7463 f7464 f7465 f7466 f7467 f7468 f7469 f747 f7470 f7471 f7472 f7473 f7474 f7475 f7476 f7477 f7478 f7479 f748 f7480 f7481 f7482 f7483 f7484 f7485 f7486 f7487 f7488 f7489 f749 f7490 f7491 f7492 f7493 f7494 f7495 f7496 f7497 f7498 f7499 f75 f750 f7500 f7501 f7502 f7503 f7504 f7505 f7506 f7507 f7508 f7509 f751 f7510 f7511 f7512 f7513 f7514 f7515 f7516 f7517 f7518 f7519 f752 f7520 f7521 f7522 f7523 f7524 f7525 f7526 f7527 f7528 f7529 f753 f7530 f7531 f7532 f7533 f7534 f7535 f7536 f7537 f7538 f7539 f754 f7540 f7541 f7542 f7543 f7544 f7545 f7546 f7547 f7548 f7549 f755 f7550 f7551 f7552 f7553 f7554 f7555 f7556 f7557 f7558 f7559 f756 f7560 f7561 f7562 f7563 f7564 f7565 f7566 f7567 f7568 f7569 f757 f7570 f7571 f7572 f7573 f7574 f7575 f7576 f7577 f7578 f7579 f758 f7580 f7581 f7582 f7583 f7584 f7585 f7586 f7587 f7588 f7589 f759 f7590 f7591 f7592 f7593 f7594 f7595 f7596 f7597 f7598 f7599 f76 f760 f7600 f7601 f7602 f7603 f7604 f7605 f7606 f7607 f7608 f7609 f761 f7610 f7611 f7612 f7613 f7614 f7615 f7616 f7617 f7618 f7619 f762 f7620 f7621 f7622 f7623 f7624 f7625 f7626 f7627 f7628 f7629 f763 f7630 f7631 f7632 f7633 f7634 f7635 f7636 f7637 f7638 f7639 f764 f7640 f7641 f7642 f7643 f7644 f7645 f7646 f7647 f7648 f7649 f765 f7650 f7651 f7652 f7653 f7654 f7655 f7656 f7657 f7658 f7659 f766 f7660 f7661 f7662 f7663 f7664 f7665 f7666 f7667 f7668 f7669 f767 f7670 f7671 f7672 f7673 f7674 f7675 f7676 f7677 f7678 f7679 f768 f7680 f7681 f7682 f7683 f7684 f7685 f7686 f7687 f7688 f7689 f769 f7690 f7691 f7692 f7693 f7694 f7695 f7696 f7697 f7698 f7699 f77 f770 f7700 f7701 f7702 f7703 f7704 f7705 f7706 f7707 f7708 f7709 f771 f7710 f7711 f7712 f7713 f7714 f7715 f7716 f7717 f7718 f7719 f772 f7720 f7721 f7722 f7723 f7724 f7725 f7726 f7727 f7728 f7729 f773 f7730 f7731 f7732 f7733 f7734 f7735 f7736 f7737 f7738 f7739 f774 f7740 f7741 f7742 f7743 f7744 f7745 f7746 f7747 f7748 f7749 f775 f7750 f7751 f7752 f7753 f7754 f7755 f7756 f7757 f7758 f7759 f776 f7760 f7761 f7762 f7763 f7764 f7765 f7766 f7767 f7768 f7769 f777 f7770 f7771 f7772 f7773 f7774 f7775 f7776 f7777 f7778 f7779 f778 f7780 f7781 f7782 f7783 f7784 f7785 f7786 f7787 f7788 f7789 f779 f7790 f7791 f7792 f7793 f7794 f7795 f7796 f7797 f7798 f7799 f78 f780 f7800 f7801 f7802 f7803 f7804 f7805 f7806 f7807 f7808 f7809 f781 f7810 f7811 f7812 f7813 f7814 f7815 f7816 f7817 f7818 f7819 f782 f7820 f7821 f7822 f7823 f7824 f7825 f7826 f7827 f7828 f7829 f783 f7830 f7831 f7832 f7833 f7834 f7835 f7836 f7837 f7838 f7839 f784 f7840 f7841 f7842 f7843 f7844 f7845 f7846 f7847 f7848 f7849 f785 f7850 f7851 f7852 f7853 f7854 f7855 f7856 f7857 f7858 f7859 f786 f7860 f7861 f7862 f7863 f7864 f7865 f7866 f7867 f7868 f7869 f787 f7870 f7871 f7872 f7873 f7874 f7875 f7876 f7877 f7878 f7879 f788 f7880 f7881 f7882 f7883 f7884 f7885 f7886 f7887 f7888 f7889 f789 f7890 f7891 f7892 f7893 f7894 f7895 f7896 f7897 f7898 f7899 f79 f790 f7900 f7901 f7902 f7903 f7904 f7905 f7906 f7907 f7908 f7909 f791 f7910 f7911 f7912 f7913 f7914 f7915 f7916 f7917 f7918 f7919 f792 f7920 f7921 f7922 f7923 f7924 f7925 f7926 f7927 f7928 f7929 f793 f7930 f7931 f7932 f7933 f7934 f7935 f7936 f7937 f7938 f7939 f794 f7940 f7941 f7942 f7943 f7944 f7945 f7946 f7947 f7948 f7949 f795 f7950 f7951 f7952 f7953 f7954 f7955 f7956 f7957 f7958 f7959 f796 f7960 f7961 f7962 f7963 f7964 f7965 f7966 f7967 f7968 f7969 f797 f7970 f7971 f7972 f7973 f7974 f7975 f7976 f7977 f7978 f7979 f798 f7980 f7981 f7982 f7983 f7984 f7985 f7986 f7987 f7988 f7989 f799 f7990 f7991 f7992 f7993 f7994 f7995 f7996 f7997 f7998 f7999 f8 f80 f800 f8000 f8001 f8002 f8003 f8004 f8005 f8006 f8007 f8008 f8009 f801 f8010 f8011 f8012 f8013 f8014 f8015 f8016 f8017 f8018 f8019 f802 f8020 f8021 f8022 f8023 f8024 f8025 f8026 f8027 f8028 f8029 f803 f8030 f8031 f8032 f8033 f8034 f8035 f8036 f8037 f8038 f8039 f804 f8040 f8041 f8042 f8043 f8044 f8045 f8046 f8047 f8048 f8049 f805 f8050 f8051 f8052 f8053 f8054 f8055 f8056 f8057 f8058 f8059 f806 f8060 f8061 f8062 f8063 f8064 f8065 f8066 f8067 f8068 f8069 f807 f8070 f8071 f8072 f8073 f8074 f8075 f8076 f8077 f8078 f8079 f808 f8080 f8081 f8082 f8083 f8084 f8085 f8086 f8087 f8088 f8089 f809 f8090 f8091 f8092 f8093 f8094 f8095 f8096 f8097 f8098 f8099 f81 f810 f8100 f8101 f8102 f8103 f8104 f8105 f8106 f8107 f8108 f8109 f811 f8110 f8111 f8112 f8113 f8114 f8115 f8116 f8117 f8118 f8119 f812 f8120 f8121 f8122 f8123 f8124 f8125 f8126 f8127 f8128 f8129 f813 f8130 f8131 f8132 f8133 f8134 f8135 f8136 f8137 f8138 f8139 f814 f8140 f8141 f8142 f8143 f8144 f8145 f8146 f8147 f8148 f8149 f815 f8150 f8151 f8152 f8153 f8154 f8155 f8156 f8157 f8158 f8159 f816 f8160 f8161 f8162 f8163 f8164 f8165 f8166 f8167 f8168 f8169 f817 f8170 f8171 f8172 f8173 f8174 f8175 f8176 f8177 f8178 f8179 f818 f8180 f8181 f8182 f8183 f8184 f8185 f8186 f8187 f8188 f8189 f819 f8190 f8191 f8192 f8193 f8194 f8195 f8196 f8197 f8198 f8199 f82 f820 f8200 f8201 f8202 f8203 f8204 f8205 f8206 f8207 f8208 f8209 f821 f8210 f8211 f8212 f8213 f8214 f8215 f8216 f8217 f8218 f8219 f822 f8220 f8221 f8222 f8223 f8224 f8225 f8226 f8227 f8228 f8229 f823 f8230 f8231 f8232 f8233 f8234 f8235 f8236 f8237 f8238 f8239 f824 f8240 f8241 f8242 f8243 f8244 f8245 f8246 f8247 f8248 f8249 f825 f8250 f8251 f8252 f8253 f8254 f8255 f8256 f8257 f8258 f8259 f826 f8260 f8261 f8262 f8263 f8264 f8265 f8266 f8267 f8268 f8269 f827 f8270 f8271 f8272 f8273 f8274 f8275 f8276 f8277 f8278 f8279 f828 f8280 f8281 f8282 f8283 f8284 f8285 f8286 f8287 f8288 f8289 f829 f8290 f8291 f8292 f8293 f8294 f8295 f8296 f8297 f8298 f8299 f83 f830 f8300 f8301 f8302 f8303 f8304 f8305 f8306 f8307 f8308 f8309 f831 f8310 f8311 f8312 f8313 f8314 f8315 f8316 f8317 f8318 f8319 f832 f8320 f8321 f8322 f8323 f8324 f8325 f8326 f8327 f8328 f8329 f833 f8330 f8331 f8332 f8333 f8334 f8335 f8336 f8337 f8338 f8339 f834 f8340 f8341 f8342 f8343 f8344 f8345 f8346 f8347 f8348 f8349 f835 f8350 f8351 f8352 f8353 f8354 f8355 f8356 f8357 f8358 f8359 f836 f8360 f8361 f8362 f8363 f8364 f8365 f8366 f8367 f8368 f8369 f837 f8370 f8371 f8372 f8373 f8374 f8375 f8376 f8377 f8378 f8379 f838 f8380 f8381 f8382 f8383 f8384 f8385 f8386 f8387 f8388 f8389 f839 f8390 f8391 f8392 f8393 f8394 f8395 f8396 f8397 f8398 f8399 f84 f840 f8400 f8401 f8402 f8403 f8404 f8405 f8406 f8407 f8408 f8409 f841 f8410 f8411 f8412 f8413 f8414 f8415 f8416 f8417 f8418 f8419 f842 f8420 f8421 f8422 f8423 f8424 f8425 f8426 f8427 f8428 f8429 f843 f8430 f8431 f8432 f8433 f8434 f8435 f8436 f8437 f8438 f8439 f844 f8440 f8441 f8442 f8443 f8444 f8445 f8446 f8447 f8448 f8449 f845 f8450 f8451 f8452 f8453 f8454 f8455 f8456 f8457 f8458 f8459 f846 f8460 f8461 f8462 f8463 f8464 f8465 f8466 f8467 f8468 f8469 f847 f8470 f8471 f8472 f8473 f8474 f8475 f8476 f8477 f8478 f8479 f848 f8480 f8481 f8482 f8483 f8484 f8485 f8486 f8487 f8488 f8489 f849 f8490 f8491 f8492 f8493 f8494 f8495 f8496 f8497 f8498 f8499 f85 f850 f8500 f8501 f8502 f8503 f8504 f8505 f8506 f8507 f8508 f8509 f851 f8510 f8511 f8512 f8513 f8514 f8515 f8516 f8517 f8518 f8519 f852 f8520 f8521 f8522 f8523 f8524 f8525 f8526 f8527 f8528 f8529 f853 f8530 f8531 f8532 f8533 f8534 f8535 f8536 f8537 f8538 f8539 f854 f8540 f8541 f8542 f8543 f8544 f8545 f8546 f8547 f8548 f8549 f855 f8550 f8551 f8552 f8553 f8554 f8555 f8556 f8557 f8558 f8559 f856 f8560 f8561 f8562 f8563 f8564 f8565 f8566 f8567 f8568 f8569 f857 f8570 f8571 f8572 f8573 f8574 f8575 f8576 f8577 f8578 f8579 f858 f8580 f8581 f8582 f8583 f8584 f8585 f8586 f8587 f8588 f8589 f859 f8590 f8591 f8592 f8593 f8594 f8595 f8596 f8597 f8598 f8599 f86 f860 f8600 f8601 f8602 f8603 f8604 f8605 f8606 f8607 f8608 f8609 f861 f8610 f8611 f8612 f8613 f8614 f8615 f8616 f8617 f8618 f8619 f862 f8620 f8621 f8622 f8623 f8624 f8625 f8626 f8627 f8628 f8629 f863 f8630 f8631 f8632 f8633 f8634 f8635 f8636 f8637 f8638 f8639 f864 f8640 f8641 f8642 f8643 f8644 f8645 f8646 f8647 f8648 f8649 f865 f8650 f8651 f8652 f8653 f8654 f8655 f8656 f8657 f8658 f8659 f866 f8660 f8661 f8662 f8663 f8664 f8665 f8666 f8667 f8668 f8669 f867 f8670 f8671 f8672 f8673 f8674 f8675 f8676 f8677 f8678 f8679 f868 f8680 f8681 f8682 f8683 f8684 f8685 f8686 f8687 f8688 f8689 f869 f8690 f8691 f8692 f8693 f8694 f8695 f8696 f8697 f8698 f8699 f87 f870 f8700 f8701 f8702 f8703 f8704 f8705 f8706 f8707 f8708 f8709 f871 f8710 f8711 f8712 f8713 f8714 f8715 f8716 f8717 f8718 f8719 f872 f8720 f8721 f8722 f8723 f8724 f8725 f8726 f8727 f8728 f8729 f873 f8730 f8731 f8732 f8733 f8734 f8735 f8736 f8737 f8738 f8739 f874 f8740 f8741 f8742 f8743 f8744 f8745 f8746 f8747 f8748 f8749 f875 f8750 f8751 f8752 f8753 f8754 f8755 f8756 f8757 f8758 f8759 f876 f8760 f8761 f8762 f8763 f8764 f8765 f8766 f8767 f8768 f8769 f877 f8770 f8771 f8772 f8773 f8774 f8775 f8776 f8777 f8778 f8779 f878 f8780 f8781 f8782 f8783 f8784 f8785 f8786 f8787 f8788 f8789 f879 f8790 f8791 f8792 f8793 f8794 f8795 f8796 f8797 f8798 f8799 f88 f880 f8800 f8801 f8802 f8803 f8804 f8805 f8806 f8807 f8808 f8809 f881 f8810 f8811 f8812 f8813 f8814 f8815 f8816 f8817 f8818 f8819 f882 f8820 f8821 f8822 f8823 f8824 f8825 f8826 f8827 f8828 f8829 f883 f8830 f8831 f8832 f8833 f8834 f8835 f8836 f8837 f8838 f8839 f884 f8840 f8841 f8842 f8843 f8844 f8845 f8846 f8847 f8848 f8849 f885 f8850 f8851 f8852 f8853 f8854 f8855 f8856 f8857 f8858 f8859 f886 f8860 f8861 f8862 f8863 f8864 f8865 f8866 f8867 f8868 f8869 f887 f8870 f8871 f8872 f8873 f8874 f8875 f8876 f8877 f8878 f8879 f888 f8880 f8881 f8882 f8883 f8884 f8885 f8886 f8887 f8888 f8889 f889 f8890 f8891 f8892 f8893 f8894 f8895 f8896 f8897 f8898 f8899 f89 f890 f8900 f8901 f8902 f8903 f8904 f8905 f8906 f8907 f8908 f8909 f891 f8910 f8911 f8912 f8913 f8914 f8915 f8916 f8917 f8918 f8919 f892 f8920 f8921 f8922 f8923 f8924 f8925 f8926 f8927 f8928 f8929 f893 f8930 f8931 f8932 f8933 f8934 f8935 f8936 f8937 f8938 f8939 f894 f8940 f8941 f8942 f8943 f8944 f8945 f8946 f8947 f8948 f8949 f895 f8950 f8951 f8952 f8953 f8954 f8955 f8956 f8957 f8958 f8959 f896 f8960 f8961 f8962 f8963 f8964 f8965 f8966 f8967 f8968 f8969 f897 f8970 f8971 f8972 f8973 f8974 f8975 f8976 f8977 f8978 f8979 f898 f8980 f8981 f8982 f8983 f8984 f8985 f8986 f8987 f8988 f8989 f899 f8990 f8991 f8992 f8993 f8994 f8995 f8996 f8997 f8998 f8999 f9 f90 f900 f9000 f9001 f9002 f9003 f9004 f9005 f9006 f9007 f9008 f9009 f901 f9010 f9011 f9012 f9013 f9014 f9015 f9016 f9017 f9018 f9019 f902 f9020 f9021 f9022 f9023 f9024 f9025 f9026 f9027 f9028 f9029 f903 f9030 f9031 f9032 f9033 f9034 f9035 f9036 f9037 f9038 f9039 f904 f9040 f9041 f9042 f9043 f9044 f9045 f9046 f9047 f9048 f9049 f905 f9050 f9051 f9052 f9053 f9054 f9055 f9056 f9057 f9058 f9059 f906 f9060 f9061 f9062 f9063 f9064 f9065 f9066 f9067 f9068 f9069 f907 f9070 f9071 f9072 f9073 f9074 f9075 f9076 f9077 f9078 f9079 f908 f9080 f9081 f9082 f9083 f9084 f9085 f9086 f9087 f9088 f9089 f909 f9090 f9091 f9092 f9093 f9094 f9095 f9096 f9097 f9098 f9099 f91 f910 f9100 f9101 f9102 f9103 f9104 f9105 f9106 f9107 f9108 f9109 f911 f9110 f9111 f9112 f9113 f9114 f9115 f9116 f9117 f9118 f9119 f912 f9120 f9121 f9122 f9123 f9124 f9125 f9126 f9127 f9128 f9129 f913 f9130 f9131 f9132 f9133 f9134 f9135 f9136 f9137 f9138 f9139 f914 f9140 f9141 f9142 f9143 f9144 f9145 f9146 f9147 f9148 f9149 f915 f9150 f9151 f9152 f9153 f9154 f9155 f9156 f9157 f9158 f9159 f916 f9160 f9161 f9162 f9163 f9164 f9165 f9166 f9167 f9168 f9169 f917 f9170 f9171 f9172 f9173 f9174 f9175 f9176 f9177 f9178 f9179 f918 f9180 f9181 f9182 f9183 f9184 f9185 f9186 f9187 f9188 f9189 f919 f9190 f9191 f9192 f9193 f9194 f9195 f9196 f9197 f9198 f9199 f92 f920 f9200 f9201 f9202 f9203 f9204 f9205 f9206 f9207 f9208 f9209 f921 f9210 f9211 f9212 f9213 f9214 f9215 f9216 f9217 f9218 f9219 f922 f9220 f9221 f9222 f9223 f9224 f9225 f9226 f9227 f9228 f9229 f923 f9230 f9231 f9232 f9233 f9234 f9235 f9236 f9237 f9238 f9239 f924 f9240 f9241 f9242 f9243 f9244 f9245 f9246 f9247 f9248 f9249 f925 f9250 f9251 f9252 f9253 f9254 f9255 f9256 f9257 f9258 f9259 f926 f9260 f9261 f9262 f9263 f9264 f9265 f9266 f9267 f9268 f9269 f927 f9270 f9271 f9272 f9273 f9274 f9275 f9276 f9277 f9278 f9279 f928 f9280 f9281 f9282 f9283 f9284 f9285 f9286 f9287 f9288 f9289 f929 f9290 f9291 f9292 f9293 f9294 f9295 f9296 f9297 f9298 f9299 f93 f930 f9300 f9301 f9302 f9303 f9304 f9305 f9306 f9307 f9308 f9309 f931 f9310 f9311 f9312 f9313 f9314 f9315 f9316 f9317 f9318 f9319 f932 f9320 f9321 f9322 f9323 f9324 f9325 f9326 f9327 f9328 f9329 f933 f9330 f9331 f9332 f9333 f9334 f9335 f9336 f9337 f9338 f9339 f934 f9340 f9341 f9342 f9343 f9344 f9345 f9346 f9347 f9348 f9349 f935 f9350 f9351 f9352 f9353 f9354 f9355 f9356 f9357 f9358 f9359 f936 f9360 f9361 f9362 f9363 f9364 f9365 f9366 f9367 f9368 f9369 f937 f9370 f9371 f9372 f9373 f9374 f9375 f9376 f9377 f9378 f9379 f938 f9380 f9381 f9382 f9383 f9384 f9385 f9386 f9387 f9388 f9389 f939 f9390 f9391 f9392 f9393 f9394 f9395 f9396 f9397 f9398 f9399 f94 f940 f9400 f9401 f9402 f9403 f9404 f9405 f9406 f9407 f9408 f9409 f941 f9410 f9411 f9412 f9413 f9414 f9415 f9416 f9417 f9418 f9419 f942 f9420 f9421 f9422 f9423 f9424 f9425 f9426 f9427 f9428 f9429 f943 f9430 f9431 f9432 f9433 f9434 f9435 f9436 f9437 f9438 f9439 f944 f9440 f9441 f9442 f9443 f9444 f9445 f9446 f9447 f9448 f9449 f945 f9450 f9451 f9452 f9453 f9454 f9455 f9456 f9457 f9458 f9459 f946 f9460 f9461 f9462 f9463 f9464 f9465 f9466 f9467 f9468 f9469 f947 f9470 f9471 f9472 f9473 f9474 f9475 f9476 f9477 f9478 f9479 f948 f9480 f9481 f9482 f9483 f9484 f9485 f9486 f9487 f9488 f9489 f949 f9490 f9491 f9492 f9493 f9494 f9495 f9496 f9497 f9498 f9499 f95 f950 f9500 f9501 f9502 f9503 f9504 f9505 f9506 f9507 f9508 f9509 f951 f9510 f9511 f9512 f9513 f9514 f9515 f9516 f9517 f9518 f9519 f952 f9520 f9521 f9522 f9523 f9524 f9525 f9526 f9527 f9528 f9529 f953 f9530 f9531 f9532 f9533 f9534 f9535 f9536 f9537 f9538 f9539 f954 f9540 f9541 f9542 f9543 f9544 f9545 f9546 f9547 f9548 f9549 f955 f9550 f9551 f9552 f9553 f9554 f9555 f9556 f9557 f9558 f9559 f956 f9560 f9561 f9562 f9563 f9564 f9565 f9566 f9567 f9568 f9569 f957 f9570 f9571 f9572 f9573 f9574 f9575 f9576 f9577 f9578 f9579 f958 f9580 f9581 f9582 f9583 f9584 f9585 f9586 f9587 f9588 f9589 f959 f9590 f9591 f9592 f9593 f9594 f9595 f9596 f9597 f9598 f9599 f96 f960 f9600 f9601 f9602 f9603 f9604 f9605 f9606 f9607 f9608 f9609 f961 f9610 f9611 f9612 f9613 f9614 f9615 f9616 f9617 f9618 f9619 f962 f9620 f9621 f9622 f9623 f9624 f9625 f9626 f9627 f9628 f9629 f963 f9630 f9631 f9632 f9633 f9634 f9635 f9636 f9637 f9638 f9639 f964 f9640 f9641 f9642 f9643 f9644 f9645 f9646 f9647 f9648 f9649 f965 f9650 f9651 f9652 f9653 f9654 f9655 f9656 f9657 f9658 f9659 f966 f9660 f9661 f9662 f9663 f9664 f9665 f9666 f9667 f9668 f9669 f967 f9670 f9671 f9672 f9673 f9674 f9675 f9676 f9677 f9678 f9679 f968 f9680 f9681 f9682 f9683 f9684 f9685 f9686 f9687 f9688 f9689 f969 f9690 f9691 f9692 f9693 f9694 f9695 f9696 f9697 f9698 f9699 f97 f970 f9700 f9701 f9702 f9703 f9704 f9705 f9706 f9707 f9708 f9709 f971 f9710 f9711 f9712 f9713 f9714 f9715 f9716 f9717 f9718 f9719 f972 f9720 f9721 f9722 f9723 f9724 f9725 f9726 f9727 f9728 f9729 f973 f9730 f9731 f9732 f9733 f9734 f9735 f9736 f9737 f9738 f9739 f974 f9740 f9741 f9742 f9743 f9744 f9745 f9746 f9747 f9748 f9749 f975 f9750 f9751 f9752 f9753 f9754 f9755 f9756 f9757 f9758 f9759 f976 f9760 f9761 f9762 f9763 f9764 f9765 f9766 f9767 f9768 f9769 f977 f9770 f9771 f9772 f9773 f9774 f9775 f9776 f9777 f9778 f9779 f978 f9780 f9781 f9782 f9783 f9784 f9785 f9786 f9787 f9788 f9789 f979 f9790 f9791 f9792 f9793 f9794 f9795 f9796 f9797 f9798 f9799 f98 f980 f9800 f9801 f9802 f9803 f9804 f9805 f9806 f9807 f9808 f9809 f981 f9810 f9811 f9812 f9813 f9814 f9815 f9816 f9817 f9818 f9819 f982 f9820 f9821 f9822 f9823 f9824 f9825 f9826 f9827 f9828 f9829 f983 f9830 f9831 f9832 f9833 f9834 f9835 f9836 f9837 f9838 f9839 f984 f9840 f9841 f9842 f9843 f9844 f9845 f9846 f9847 f9848 f9849 f985 f9850 f9851 f9852 f9853 f9854 f9855 f9856 f9857 f9858 f9859 f986 f9860 f9861 f9862 f9863 f9864 f9865 f9866 f9867 f9868 f9869 f987 f9870 f9871 f9872 f9873 f9874 f9875 f9876 f9877 f9878 f9879 f988 f9880 f9881 f9882 f9883 f9884 f9885 f9886 f9887 f9888 f9889 f989 f9890 f9891 f9892 f9893 f9894 f9895 f9896 f9897 f9898 f9899 f99 f990 f9900 f9901 f9902 f9903 f9904 f9905 f9906 f9907 f9908 f9909 f991 f9910 f9911 f9912 f9913 f9914 f9915 f9916 f9917 f9918 f9919 f992 f9920 f9921 f9922 f9923 f9924 f9925 f9926 f9927 f9928 f9929 f993 f9930 f9931 f9932 f9933 f9934 f9935 f9936 f9937 f9938 f9939 f994 f9940 f9941 f9942 f9943 f9944 f9945 f9946 f9947 f9948 f9949 f995 f9950 f9951 f9952 f9953 f9954 f9955 f9956 f9957 f9958 f9959 f996 f9960 f9961 f9962 f9963 f9964 f9965 f9966 f9967 f9968 f9969 f997 f9970 f9971 f9972 f9973 f9974 f9975 f9976 f9977 f9978 f9979 f998 f9980 f9981 f9982 f9983 f9984 f9985 f9986 f9987 f9988 f9989 f999 f9990 f9991 f9992 f9993 f9994 f9995 f9996 f9997 f9998 f9999 ./d40d.sanityn: ./d43j.sanityn: ./d60.sanityn: file ./d70a.sanityn: ./d70b.sanityn: ./d71b.sanityn: f71b.sanityn ./d71c.sanityn: ./d77m.sanityn: nrs_delay_oleg135-client.virtnet ./d79.sanityn: ./d80b.sanityn: file1 file2 file3 file4 file5 migrate_dir ./d80b.sanityn/migrate_dir: file1 file2 file3 file4 file5 ./d81a.sanityn: d0 d1 ./d81a.sanityn/d0: 0 ./d81a.sanityn/d1: ./d81b.sanityn: f81b.sanityn-new.0 f81b.sanityn-new.1 f81b.sanityn-new.10 f81b.sanityn-new.100 f81b.sanityn-new.1000 f81b.sanityn-new.1001 f81b.sanityn-new.1002 f81b.sanityn-new.1003 f81b.sanityn-new.1004 f81b.sanityn-new.1005 f81b.sanityn-new.1006 f81b.sanityn-new.1007 f81b.sanityn-new.1008 f81b.sanityn-new.1009 f81b.sanityn-new.101 f81b.sanityn-new.1010 f81b.sanityn-new.1011 f81b.sanityn-new.1012 f81b.sanityn-new.1013 f81b.sanityn-new.1014 f81b.sanityn-new.1015 f81b.sanityn-new.1016 f81b.sanityn-new.1017 f81b.sanityn-new.1018 f81b.sanityn-new.1019 f81b.sanityn-new.102 f81b.sanityn-new.1020 f81b.sanityn-new.1021 f81b.sanityn-new.1022 f81b.sanityn-new.1023 f81b.sanityn-new.1024 f81b.sanityn-new.1025 f81b.sanityn-new.1026 f81b.sanityn-new.1027 f81b.sanityn-new.1028 f81b.sanityn-new.1029 f81b.sanityn-new.103 f81b.sanityn-new.1030 f81b.sanityn-new.1031 f81b.sanityn-new.1032 f81b.sanityn-new.1033 f81b.sanityn-new.1034 f81b.sanityn-new.1035 f81b.sanityn-new.1036 f81b.sanityn-new.1037 f81b.sanityn-new.1038 f81b.sanityn-new.1039 f81b.sanityn-new.104 f81b.sanityn-new.1040 f81b.sanityn-new.1041 f81b.sanityn-new.1042 f81b.sanityn-new.1043 f81b.sanityn-new.1044 f81b.sanityn-new.1045 f81b.sanityn-new.1046 f81b.sanityn-new.1047 f81b.sanityn-new.1048 f81b.sanityn-new.1049 f81b.sanityn-new.105 f81b.sanityn-new.1050 f81b.sanityn-new.1051 f81b.sanityn-new.1052 f81b.sanityn-new.1053 f81b.sanityn-new.1054 f81b.sanityn-new.1055 f81b.sanityn-new.1056 f81b.sanityn-new.1057 f81b.sanityn-new.1058 f81b.sanityn-new.1059 f81b.sanityn-new.106 f81b.sanityn-new.1060 f81b.sanityn-new.1061 f81b.sanityn-new.1062 f81b.sanityn-new.1063 f81b.sanityn-new.1064 f81b.sanityn-new.1065 f81b.sanityn-new.1066 f81b.sanityn-new.1067 f81b.sanityn-new.1068 f81b.sanityn-new.1069 f81b.sanityn-new.107 f81b.sanityn-new.1070 f81b.sanityn-new.1071 f81b.sanityn-new.1072 f81b.sanityn-new.1073 f81b.sanityn-new.1074 f81b.sanityn-new.1075 f81b.sanityn-new.1076 f81b.sanityn-new.1077 f81b.sanityn-new.1078 f81b.sanityn-new.1079 f81b.sanityn-new.108 f81b.sanityn-new.1080 f81b.sanityn-new.1081 f81b.sanityn-new.1082 f81b.sanityn-new.1083 f81b.sanityn-new.1084 f81b.sanityn-new.1085 f81b.sanityn-new.1086 f81b.sanityn-new.1087 f81b.sanityn-new.1088 f81b.sanityn-new.1089 f81b.sanityn-new.109 f81b.sanityn-new.1090 f81b.sanityn-new.1091 f81b.sanityn-new.1092 f81b.sanityn-new.1093 f81b.sanityn-new.1094 f81b.sanityn-new.1095 f81b.sanityn-new.1096 f81b.sanityn-new.1097 f81b.sanityn-new.1098 f81b.sanityn-new.1099 f81b.sanityn-new.11 f81b.sanityn-new.110 f81b.sanityn-new.1100 f81b.sanityn-new.1101 f81b.sanityn-new.1102 f81b.sanityn-new.1103 f81b.sanityn-new.1104 f81b.sanityn-new.1105 f81b.sanityn-new.1106 f81b.sanityn-new.1107 f81b.sanityn-new.1108 f81b.sanityn-new.1109 f81b.sanityn-new.111 f81b.sanityn-new.1110 f81b.sanityn-new.1111 f81b.sanityn-new.1112 f81b.sanityn-new.1113 f81b.sanityn-new.1114 f81b.sanityn-new.1115 f81b.sanityn-new.1116 f81b.sanityn-new.1117 f81b.sanityn-new.1118 f81b.sanityn-new.1119 f81b.sanityn-new.112 f81b.sanityn-new.1120 f81b.sanityn-new.1121 f81b.sanityn-new.1122 f81b.sanityn-new.1123 f81b.sanityn-new.1124 f81b.sanityn-new.1125 f81b.sanityn-new.1126 f81b.sanityn-new.1127 f81b.sanityn-new.1128 f81b.sanityn-new.1129 f81b.sanityn-new.113 f81b.sanityn-new.1130 f81b.sanityn-new.1131 f81b.sanityn-new.1132 f81b.sanityn-new.1133 f81b.sanityn-new.1134 f81b.sanityn-new.1135 f81b.sanityn-new.1136 f81b.sanityn-new.1137 f81b.sanityn-new.1138 f81b.sanityn-new.1139 f81b.sanityn-new.114 f81b.sanityn-new.1140 f81b.sanityn-new.1141 f81b.sanityn-new.1142 f81b.sanityn-new.1143 f81b.sanityn-new.1144 f81b.sanityn-new.1145 f81b.sanityn-new.1146 f81b.sanityn-new.1147 f81b.sanityn-new.1148 f81b.sanityn-new.1149 f81b.sanityn-new.115 f81b.sanityn-new.1150 f81b.sanityn-new.1151 f81b.sanityn-new.1152 f81b.sanityn-new.1153 f81b.sanityn-new.1154 f81b.sanityn-new.1155 f81b.sanityn-new.1156 f81b.sanityn-new.1157 f81b.sanityn-new.1158 f81b.sanityn-new.1159 f81b.sanityn-new.116 f81b.sanityn-new.1160 f81b.sanityn-new.1161 f81b.sanityn-new.1162 f81b.sanityn-new.1163 f81b.sanityn-new.1164 f81b.sanityn-new.1165 f81b.sanityn-new.1166 f81b.sanityn-new.1167 f81b.sanityn-new.1168 f81b.sanityn-new.1169 f81b.sanityn-new.117 f81b.sanityn-new.1170 f81b.sanityn-new.1171 f81b.sanityn-new.1172 f81b.sanityn-new.1173 f81b.sanityn-new.1174 f81b.sanityn-new.1175 f81b.sanityn-new.1176 f81b.sanityn-new.1177 f81b.sanityn-new.1178 f81b.sanityn-new.1179 f81b.sanityn-new.118 f81b.sanityn-new.1180 f81b.sanityn-new.1181 f81b.sanityn-new.1182 f81b.sanityn-new.1183 f81b.sanityn-new.1184 f81b.sanityn-new.1185 f81b.sanityn-new.1186 f81b.sanityn-new.1187 f81b.sanityn-new.1188 f81b.sanityn-new.1189 f81b.sanityn-new.119 f81b.sanityn-new.1190 f81b.sanityn-new.1191 f81b.sanityn-new.1192 f81b.sanityn-new.1193 f81b.sanityn-new.1194 f81b.sanityn-new.1195 f81b.sanityn-new.1196 f81b.sanityn-new.1197 f81b.sanityn-new.1198 f81b.sanityn-new.1199 f81b.sanityn-new.12 f81b.sanityn-new.120 f81b.sanityn-new.1200 f81b.sanityn-new.1201 f81b.sanityn-new.1202 f81b.sanityn-new.1203 f81b.sanityn-new.1204 f81b.sanityn-new.1205 f81b.sanityn-new.1206 f81b.sanityn-new.1207 f81b.sanityn-new.1208 f81b.sanityn-new.1209 f81b.sanityn-new.121 f81b.sanityn-new.1210 f81b.sanityn-new.1211 f81b.sanityn-new.1212 f81b.sanityn-new.1213 f81b.sanityn-new.1214 f81b.sanityn-new.1215 f81b.sanityn-new.1216 f81b.sanityn-new.1217 f81b.sanityn-new.1218 f81b.sanityn-new.1219 f81b.sanityn-new.122 f81b.sanityn-new.1220 f81b.sanityn-new.1221 f81b.sanityn-new.1222 f81b.sanityn-new.1223 f81b.sanityn-new.1224 f81b.sanityn-new.1225 f81b.sanityn-new.1226 f81b.sanityn-new.1227 f81b.sanityn-new.1228 f81b.sanityn-new.1229 f81b.sanityn-new.123 f81b.sanityn-new.1230 f81b.sanityn-new.1231 f81b.sanityn-new.1232 f81b.sanityn-new.1233 f81b.sanityn-new.1234 f81b.sanityn-new.1235 f81b.sanityn-new.1236 f81b.sanityn-new.1237 f81b.sanityn-new.1238 f81b.sanityn-new.1239 f81b.sanityn-new.124 f81b.sanityn-new.1240 f81b.sanityn-new.1241 f81b.sanityn-new.1242 f81b.sanityn-new.1243 f81b.sanityn-new.1244 f81b.sanityn-new.1245 f81b.sanityn-new.1246 f81b.sanityn-new.1247 f81b.sanityn-new.1248 f81b.sanityn-new.1249 f81b.sanityn-new.125 f81b.sanityn-new.1250 f81b.sanityn-new.1251 f81b.sanityn-new.1252 f81b.sanityn-new.1253 f81b.sanityn-new.1254 f81b.sanityn-new.1255 f81b.sanityn-new.1256 f81b.sanityn-new.1257 f81b.sanityn-new.1258 f81b.sanityn-new.1259 f81b.sanityn-new.126 f81b.sanityn-new.1260 f81b.sanityn-new.1261 f81b.sanityn-new.1262 f81b.sanityn-new.1263 f81b.sanityn-new.1264 f81b.sanityn-new.1265 f81b.sanityn-new.1266 f81b.sanityn-new.1267 f81b.sanityn-new.1268 f81b.sanityn-new.1269 f81b.sanityn-new.127 f81b.sanityn-new.1270 f81b.sanityn-new.1271 f81b.sanityn-new.1272 f81b.sanityn-new.1273 f81b.sanityn-new.1274 f81b.sanityn-new.1275 f81b.sanityn-new.1276 f81b.sanityn-new.1277 f81b.sanityn-new.1278 f81b.sanityn-new.1279 f81b.sanityn-new.128 f81b.sanityn-new.1280 f81b.sanityn-new.1281 f81b.sanityn-new.1282 f81b.sanityn-new.1283 f81b.sanityn-new.1284 f81b.sanityn-new.1285 f81b.sanityn-new.1286 f81b.sanityn-new.1287 f81b.sanityn-new.1288 f81b.sanityn-new.1289 f81b.sanityn-new.129 f81b.sanityn-new.1290 f81b.sanityn-new.1291 f81b.sanityn-new.1292 f81b.sanityn-new.1293 f81b.sanityn-new.1294 f81b.sanityn-new.1295 f81b.sanityn-new.1296 f81b.sanityn-new.1297 f81b.sanityn-new.1298 f81b.sanityn-new.1299 f81b.sanityn-new.13 f81b.sanityn-new.130 f81b.sanityn-new.1300 f81b.sanityn-new.1301 f81b.sanityn-new.1302 f81b.sanityn-new.1303 f81b.sanityn-new.1304 f81b.sanityn-new.1305 f81b.sanityn-new.1306 f81b.sanityn-new.1307 f81b.sanityn-new.1308 f81b.sanityn-new.1309 f81b.sanityn-new.131 f81b.sanityn-new.1310 f81b.sanityn-new.1311 f81b.sanityn-new.1312 f81b.sanityn-new.1313 f81b.sanityn-new.1314 f81b.sanityn-new.1315 f81b.sanityn-new.1316 f81b.sanityn-new.1317 f81b.sanityn-new.1318 f81b.sanityn-new.1319 f81b.sanityn-new.132 f81b.sanityn-new.1320 f81b.sanityn-new.1321 f81b.sanityn-new.1322 f81b.sanityn-new.1323 f81b.sanityn-new.1324 f81b.sanityn-new.1325 f81b.sanityn-new.1326 f81b.sanityn-new.1327 f81b.sanityn-new.1328 f81b.sanityn-new.1329 f81b.sanityn-new.133 f81b.sanityn-new.1330 f81b.sanityn-new.1331 f81b.sanityn-new.1332 f81b.sanityn-new.1333 f81b.sanityn-new.1334 f81b.sanityn-new.1335 f81b.sanityn-new.1336 f81b.sanityn-new.1337 f81b.sanityn-new.1338 f81b.sanityn-new.1339 f81b.sanityn-new.134 f81b.sanityn-new.1340 f81b.sanityn-new.1341 f81b.sanityn-new.1342 f81b.sanityn-new.1343 f81b.sanityn-new.1344 f81b.sanityn-new.1345 f81b.sanityn-new.1346 f81b.sanityn-new.1347 f81b.sanityn-new.1348 f81b.sanityn-new.1349 f81b.sanityn-new.135 f81b.sanityn-new.1350 f81b.sanityn-new.1351 f81b.sanityn-new.1352 f81b.sanityn-new.1353 f81b.sanityn-new.1354 f81b.sanityn-new.1355 f81b.sanityn-new.1356 f81b.sanityn-new.1357 f81b.sanityn-new.1358 f81b.sanityn-new.1359 f81b.sanityn-new.136 f81b.sanityn-new.1360 f81b.sanityn-new.1361 f81b.sanityn-new.1362 f81b.sanityn-new.1363 f81b.sanityn-new.1364 f81b.sanityn-new.1365 f81b.sanityn-new.1366 f81b.sanityn-new.1367 f81b.sanityn-new.1368 f81b.sanityn-new.1369 f81b.sanityn-new.137 f81b.sanityn-new.1370 f81b.sanityn-new.1371 f81b.sanityn-new.1372 f81b.sanityn-new.1373 f81b.sanityn-new.1374 f81b.sanityn-new.1375 f81b.sanityn-new.1376 f81b.sanityn-new.1377 f81b.sanityn-new.1378 f81b.sanityn-new.1379 f81b.sanityn-new.138 f81b.sanityn-new.1380 f81b.sanityn-new.1381 f81b.sanityn-new.1382 f81b.sanityn-new.1383 f81b.sanityn-new.1384 f81b.sanityn-new.1385 f81b.sanityn-new.1386 f81b.sanityn-new.1387 f81b.sanityn-new.1388 f81b.sanityn-new.1389 f81b.sanityn-new.139 f81b.sanityn-new.1390 f81b.sanityn-new.1391 f81b.sanityn-new.1392 f81b.sanityn-new.1393 f81b.sanityn-new.1394 f81b.sanityn-new.1395 f81b.sanityn-new.1396 f81b.sanityn-new.1397 f81b.sanityn-new.1398 f81b.sanityn-new.1399 f81b.sanityn-new.14 f81b.sanityn-new.140 f81b.sanityn-new.1400 f81b.sanityn-new.1401 f81b.sanityn-new.1402 f81b.sanityn-new.1403 f81b.sanityn-new.1404 f81b.sanityn-new.1405 f81b.sanityn-new.1406 f81b.sanityn-new.1407 f81b.sanityn-new.1408 f81b.sanityn-new.1409 f81b.sanityn-new.141 f81b.sanityn-new.1410 f81b.sanityn-new.1411 f81b.sanityn-new.1412 f81b.sanityn-new.1413 f81b.sanityn-new.1414 f81b.sanityn-new.1415 f81b.sanityn-new.1416 f81b.sanityn-new.1417 f81b.sanityn-new.1418 f81b.sanityn-new.1419 f81b.sanityn-new.142 f81b.sanityn-new.1420 f81b.sanityn-new.1421 f81b.sanityn-new.1422 f81b.sanityn-new.1423 f81b.sanityn-new.1424 f81b.sanityn-new.1425 f81b.sanityn-new.1426 f81b.sanityn-new.1427 f81b.sanityn-new.1428 f81b.sanityn-new.1429 f81b.sanityn-new.143 f81b.sanityn-new.1430 f81b.sanityn-new.1431 f81b.sanityn-new.1432 f81b.sanityn-new.1433 f81b.sanityn-new.1434 f81b.sanityn-new.1435 f81b.sanityn-new.1436 f81b.sanityn-new.1437 f81b.sanityn-new.1438 f81b.sanityn-new.1439 f81b.sanityn-new.144 f81b.sanityn-new.1440 f81b.sanityn-new.1441 f81b.sanityn-new.1442 f81b.sanityn-new.1443 f81b.sanityn-new.1444 f81b.sanityn-new.1445 f81b.sanityn-new.1446 f81b.sanityn-new.1447 f81b.sanityn-new.1448 f81b.sanityn-new.1449 f81b.sanityn-new.145 f81b.sanityn-new.1450 f81b.sanityn-new.1451 f81b.sanityn-new.1452 f81b.sanityn-new.1453 f81b.sanityn-new.1454 f81b.sanityn-new.1455 f81b.sanityn-new.1456 f81b.sanityn-new.1457 f81b.sanityn-new.1458 f81b.sanityn-new.1459 f81b.sanityn-new.146 f81b.sanityn-new.1460 f81b.sanityn-new.1461 f81b.sanityn-new.1462 f81b.sanityn-new.1463 f81b.sanityn-new.1464 f81b.sanityn-new.1465 f81b.sanityn-new.1466 f81b.sanityn-new.1467 f81b.sanityn-new.1468 f81b.sanityn-new.1469 f81b.sanityn-new.147 f81b.sanityn-new.1470 f81b.sanityn-new.1471 f81b.sanityn-new.1472 f81b.sanityn-new.1473 f81b.sanityn-new.1474 f81b.sanityn-new.1475 f81b.sanityn-new.1476 f81b.sanityn-new.1477 f81b.sanityn-new.1478 f81b.sanityn-new.1479 f81b.sanityn-new.148 f81b.sanityn-new.1480 f81b.sanityn-new.1481 f81b.sanityn-new.1482 f81b.sanityn-new.1483 f81b.sanityn-new.1484 f81b.sanityn-new.1485 f81b.sanityn-new.1486 f81b.sanityn-new.1487 f81b.sanityn-new.1488 f81b.sanityn-new.1489 f81b.sanityn-new.149 f81b.sanityn-new.1490 f81b.sanityn-new.1491 f81b.sanityn-new.1492 f81b.sanityn-new.1493 f81b.sanityn-new.1494 f81b.sanityn-new.1495 f81b.sanityn-new.1496 f81b.sanityn-new.1497 f81b.sanityn-new.1498 f81b.sanityn-new.1499 f81b.sanityn-new.15 f81b.sanityn-new.150 f81b.sanityn-new.1500 f81b.sanityn-new.1501 f81b.sanityn-new.1502 f81b.sanityn-new.1503 f81b.sanityn-new.1504 f81b.sanityn-new.1505 f81b.sanityn-new.1506 f81b.sanityn-new.1507 f81b.sanityn-new.1508 f81b.sanityn-new.1509 f81b.sanityn-new.151 f81b.sanityn-new.1510 f81b.sanityn-new.1511 f81b.sanityn-new.1512 f81b.sanityn-new.1513 f81b.sanityn-new.1514 f81b.sanityn-new.1515 f81b.sanityn-new.1516 f81b.sanityn-new.1517 f81b.sanityn-new.1518 f81b.sanityn-new.1519 f81b.sanityn-new.152 f81b.sanityn-new.1520 f81b.sanityn-new.1521 f81b.sanityn-new.1522 f81b.sanityn-new.1523 f81b.sanityn-new.1524 f81b.sanityn-new.1525 f81b.sanityn-new.1526 f81b.sanityn-new.1527 f81b.sanityn-new.1528 f81b.sanityn-new.1529 f81b.sanityn-new.153 f81b.sanityn-new.1530 f81b.sanityn-new.1531 f81b.sanityn-new.1532 f81b.sanityn-new.1533 f81b.sanityn-new.1534 f81b.sanityn-new.1535 f81b.sanityn-new.1536 f81b.sanityn-new.1537 f81b.sanityn-new.1538 f81b.sanityn-new.1539 f81b.sanityn-new.154 f81b.sanityn-new.1540 f81b.sanityn-new.1541 f81b.sanityn-new.1542 f81b.sanityn-new.1543 f81b.sanityn-new.1544 f81b.sanityn-new.1545 f81b.sanityn-new.1546 f81b.sanityn-new.1547 f81b.sanityn-new.1548 f81b.sanityn-new.1549 f81b.sanityn-new.155 f81b.sanityn-new.1550 f81b.sanityn-new.1551 f81b.sanityn-new.1552 f81b.sanityn-new.1553 f81b.sanityn-new.1554 f81b.sanityn-new.1555 f81b.sanityn-new.1556 f81b.sanityn-new.1557 f81b.sanityn-new.1558 f81b.sanityn-new.1559 f81b.sanityn-new.156 f81b.sanityn-new.1560 f81b.sanityn-new.1561 f81b.sanityn-new.1562 f81b.sanityn-new.1563 f81b.sanityn-new.1564 f81b.sanityn-new.1565 f81b.sanityn-new.1566 f81b.sanityn-new.1567 f81b.sanityn-new.1568 f81b.sanityn-new.1569 f81b.sanityn-new.157 f81b.sanityn-new.1570 f81b.sanityn-new.1571 f81b.sanityn-new.1572 f81b.sanityn-new.1573 f81b.sanityn-new.1574 f81b.sanityn-new.1575 f81b.sanityn-new.1576 f81b.sanityn-new.1577 f81b.sanityn-new.1578 f81b.sanityn-new.1579 f81b.sanityn-new.158 f81b.sanityn-new.1580 f81b.sanityn-new.1581 f81b.sanityn-new.1582 f81b.sanityn-new.1583 f81b.sanityn-new.1584 f81b.sanityn-new.1585 f81b.sanityn-new.1586 f81b.sanityn-new.1587 f81b.sanityn-new.1588 f81b.sanityn-new.1589 f81b.sanityn-new.159 f81b.sanityn-new.1590 f81b.sanityn-new.1591 f81b.sanityn-new.1592 f81b.sanityn-new.1593 f81b.sanityn-new.1594 f81b.sanityn-new.1595 f81b.sanityn-new.1596 f81b.sanityn-new.1597 f81b.sanityn-new.1598 f81b.sanityn-new.1599 f81b.sanityn-new.16 f81b.sanityn-new.160 f81b.sanityn-new.1600 f81b.sanityn-new.1601 f81b.sanityn-new.1602 f81b.sanityn-new.1603 f81b.sanityn-new.1604 f81b.sanityn-new.1605 f81b.sanityn-new.1606 f81b.sanityn-new.1607 f81b.sanityn-new.1608 f81b.sanityn-new.1609 f81b.sanityn-new.161 f81b.sanityn-new.1610 f81b.sanityn-new.1611 f81b.sanityn-new.1612 f81b.sanityn-new.1613 f81b.sanityn-new.1614 f81b.sanityn-new.1615 f81b.sanityn-new.1616 f81b.sanityn-new.1617 f81b.sanityn-new.1618 f81b.sanityn-new.1619 f81b.sanityn-new.162 f81b.sanityn-new.1620 f81b.sanityn-new.1621 f81b.sanityn-new.1622 f81b.sanityn-new.1623 f81b.sanityn-new.1624 f81b.sanityn-new.1625 f81b.sanityn-new.1626 f81b.sanityn-new.1627 f81b.sanityn-new.1628 f81b.sanityn-new.1629 f81b.sanityn-new.163 f81b.sanityn-new.1630 f81b.sanityn-new.1631 f81b.sanityn-new.1632 f81b.sanityn-new.1633 f81b.sanityn-new.1634 f81b.sanityn-new.1635 f81b.sanityn-new.1636 f81b.sanityn-new.1637 f81b.sanityn-new.1638 f81b.sanityn-new.1639 f81b.sanityn-new.164 f81b.sanityn-new.1640 f81b.sanityn-new.1641 f81b.sanityn-new.1642 f81b.sanityn-new.1643 f81b.sanityn-new.1644 f81b.sanityn-new.1645 f81b.sanityn-new.1646 f81b.sanityn-new.1647 f81b.sanityn-new.1648 f81b.sanityn-new.1649 f81b.sanityn-new.165 f81b.sanityn-new.1650 f81b.sanityn-new.1651 f81b.sanityn-new.1652 f81b.sanityn-new.1653 f81b.sanityn-new.1654 f81b.sanityn-new.1655 f81b.sanityn-new.1656 f81b.sanityn-new.1657 f81b.sanityn-new.1658 f81b.sanityn-new.1659 f81b.sanityn-new.166 f81b.sanityn-new.1660 f81b.sanityn-new.1661 f81b.sanityn-new.1662 f81b.sanityn-new.1663 f81b.sanityn-new.1664 f81b.sanityn-new.1665 f81b.sanityn-new.1666 f81b.sanityn-new.1667 f81b.sanityn-new.1668 f81b.sanityn-new.1669 f81b.sanityn-new.167 f81b.sanityn-new.1670 f81b.sanityn-new.1671 f81b.sanityn-new.1672 f81b.sanityn-new.1673 f81b.sanityn-new.1674 f81b.sanityn-new.1675 f81b.sanityn-new.1676 f81b.sanityn-new.1677 f81b.sanityn-new.1678 f81b.sanityn-new.1679 f81b.sanityn-new.168 f81b.sanityn-new.1680 f81b.sanityn-new.1681 f81b.sanityn-new.1682 f81b.sanityn-new.1683 f81b.sanityn-new.1684 f81b.sanityn-new.1685 f81b.sanityn-new.1686 f81b.sanityn-new.1687 f81b.sanityn-new.1688 f81b.sanityn-new.1689 f81b.sanityn-new.169 f81b.sanityn-new.1690 f81b.sanityn-new.1691 f81b.sanityn-new.1692 f81b.sanityn-new.1693 f81b.sanityn-new.1694 f81b.sanityn-new.1695 f81b.sanityn-new.1696 f81b.sanityn-new.1697 f81b.sanityn-new.1698 f81b.sanityn-new.1699 f81b.sanityn-new.17 f81b.sanityn-new.170 f81b.sanityn-new.1700 f81b.sanityn-new.1701 f81b.sanityn-new.1702 f81b.sanityn-new.1703 f81b.sanityn-new.1704 f81b.sanityn-new.1705 f81b.sanityn-new.1706 f81b.sanityn-new.1707 f81b.sanityn-new.1708 f81b.sanityn-new.1709 f81b.sanityn-new.171 f81b.sanityn-new.1710 f81b.sanityn-new.1711 f81b.sanityn-new.1712 f81b.sanityn-new.1713 f81b.sanityn-new.1714 f81b.sanityn-new.1715 f81b.sanityn-new.1716 f81b.sanityn-new.1717 f81b.sanityn-new.1718 f81b.sanityn-new.1719 f81b.sanityn-new.172 f81b.sanityn-new.1720 f81b.sanityn-new.1721 f81b.sanityn-new.1722 f81b.sanityn-new.1723 f81b.sanityn-new.1724 f81b.sanityn-new.1725 f81b.sanityn-new.1726 f81b.sanityn-new.1727 f81b.sanityn-new.1728 f81b.sanityn-new.1729 f81b.sanityn-new.173 f81b.sanityn-new.1730 f81b.sanityn-new.1731 f81b.sanityn-new.1732 f81b.sanityn-new.1733 f81b.sanityn-new.1734 f81b.sanityn-new.1735 f81b.sanityn-new.1736 f81b.sanityn-new.1737 f81b.sanityn-new.1738 f81b.sanityn-new.1739 f81b.sanityn-new.174 f81b.sanityn-new.1740 f81b.sanityn-new.1741 f81b.sanityn-new.1742 f81b.sanityn-new.1743 f81b.sanityn-new.1744 f81b.sanityn-new.1745 f81b.sanityn-new.1746 f81b.sanityn-new.1747 f81b.sanityn-new.1748 f81b.sanityn-new.1749 f81b.sanityn-new.175 f81b.sanityn-new.1750 f81b.sanityn-new.1751 f81b.sanityn-new.1752 f81b.sanityn-new.1753 f81b.sanityn-new.1754 f81b.sanityn-new.1755 f81b.sanityn-new.1756 f81b.sanityn-new.1757 f81b.sanityn-new.1758 f81b.sanityn-new.1759 f81b.sanityn-new.176 f81b.sanityn-new.1760 f81b.sanityn-new.1761 f81b.sanityn-new.1762 f81b.sanityn-new.1763 f81b.sanityn-new.1764 f81b.sanityn-new.1765 f81b.sanityn-new.1766 f81b.sanityn-new.1767 f81b.sanityn-new.1768 f81b.sanityn-new.1769 f81b.sanityn-new.177 f81b.sanityn-new.1770 f81b.sanityn-new.1771 f81b.sanityn-new.1772 f81b.sanityn-new.1773 f81b.sanityn-new.1774 f81b.sanityn-new.1775 f81b.sanityn-new.1776 f81b.sanityn-new.1777 f81b.sanityn-new.1778 f81b.sanityn-new.1779 f81b.sanityn-new.178 f81b.sanityn-new.1780 f81b.sanityn-new.1781 f81b.sanityn-new.1782 f81b.sanityn-new.1783 f81b.sanityn-new.1784 f81b.sanityn-new.1785 f81b.sanityn-new.1786 f81b.sanityn-new.1787 f81b.sanityn-new.1788 f81b.sanityn-new.1789 f81b.sanityn-new.179 f81b.sanityn-new.1790 f81b.sanityn-new.1791 f81b.sanityn-new.1792 f81b.sanityn-new.1793 f81b.sanityn-new.1794 f81b.sanityn-new.1795 f81b.sanityn-new.1796 f81b.sanityn-new.1797 f81b.sanityn-new.1798 f81b.sanityn-new.1799 f81b.sanityn-new.18 f81b.sanityn-new.180 f81b.sanityn-new.1800 f81b.sanityn-new.1801 f81b.sanityn-new.1802 f81b.sanityn-new.1803 f81b.sanityn-new.1804 f81b.sanityn-new.1805 f81b.sanityn-new.1806 f81b.sanityn-new.1807 f81b.sanityn-new.1808 f81b.sanityn-new.1809 f81b.sanityn-new.181 f81b.sanityn-new.1810 f81b.sanityn-new.1811 f81b.sanityn-new.1812 f81b.sanityn-new.1813 f81b.sanityn-new.1814 f81b.sanityn-new.1815 f81b.sanityn-new.1816 f81b.sanityn-new.1817 f81b.sanityn-new.1818 f81b.sanityn-new.1819 f81b.sanityn-new.182 f81b.sanityn-new.1820 f81b.sanityn-new.1821 f81b.sanityn-new.1822 f81b.sanityn-new.1823 f81b.sanityn-new.1824 f81b.sanityn-new.1825 f81b.sanityn-new.1826 f81b.sanityn-new.1827 f81b.sanityn-new.1828 f81b.sanityn-new.1829 f81b.sanityn-new.183 f81b.sanityn-new.1830 f81b.sanityn-new.1831 f81b.sanityn-new.1832 f81b.sanityn-new.1833 f81b.sanityn-new.1834 f81b.sanityn-new.1835 f81b.sanityn-new.1836 f81b.sanityn-new.1837 f81b.sanityn-new.1838 f81b.sanityn-new.1839 f81b.sanityn-new.184 f81b.sanityn-new.1840 f81b.sanityn-new.1841 f81b.sanityn-new.1842 f81b.sanityn-new.1843 f81b.sanityn-new.1844 f81b.sanityn-new.1845 f81b.sanityn-new.1846 f81b.sanityn-new.1847 f81b.sanityn-new.1848 f81b.sanityn-new.1849 f81b.sanityn-new.185 f81b.sanityn-new.1850 f81b.sanityn-new.1851 f81b.sanityn-new.1852 f81b.sanityn-new.1853 f81b.sanityn-new.1854 f81b.sanityn-new.1855 f81b.sanityn-new.1856 f81b.sanityn-new.1857 f81b.sanityn-new.1858 f81b.sanityn-new.1859 f81b.sanityn-new.186 f81b.sanityn-new.1860 f81b.sanityn-new.1861 f81b.sanityn-new.1862 f81b.sanityn-new.1863 f81b.sanityn-new.1864 f81b.sanityn-new.1865 f81b.sanityn-new.1866 f81b.sanityn-new.1867 f81b.sanityn-new.1868 f81b.sanityn-new.1869 f81b.sanityn-new.187 f81b.sanityn-new.1870 f81b.sanityn-new.1871 f81b.sanityn-new.1872 f81b.sanityn-new.1873 f81b.sanityn-new.1874 f81b.sanityn-new.1875 f81b.sanityn-new.1876 f81b.sanityn-new.1877 f81b.sanityn-new.1878 f81b.sanityn-new.1879 f81b.sanityn-new.188 f81b.sanityn-new.1880 f81b.sanityn-new.1881 f81b.sanityn-new.1882 f81b.sanityn-new.1883 f81b.sanityn-new.1884 f81b.sanityn-new.1885 f81b.sanityn-new.1886 f81b.sanityn-new.1887 f81b.sanityn-new.1888 f81b.sanityn-new.1889 f81b.sanityn-new.189 f81b.sanityn-new.1890 f81b.sanityn-new.1891 f81b.sanityn-new.1892 f81b.sanityn-new.1893 f81b.sanityn-new.1894 f81b.sanityn-new.1895 f81b.sanityn-new.1896 f81b.sanityn-new.1897 f81b.sanityn-new.1898 f81b.sanityn-new.1899 f81b.sanityn-new.19 f81b.sanityn-new.190 f81b.sanityn-new.1900 f81b.sanityn-new.1901 f81b.sanityn-new.1902 f81b.sanityn-new.1903 f81b.sanityn-new.1904 f81b.sanityn-new.1905 f81b.sanityn-new.1906 f81b.sanityn-new.1907 f81b.sanityn-new.1908 f81b.sanityn-new.1909 f81b.sanityn-new.191 f81b.sanityn-new.1910 f81b.sanityn-new.1911 f81b.sanityn-new.1912 f81b.sanityn-new.1913 f81b.sanityn-new.1914 f81b.sanityn-new.1915 f81b.sanityn-new.1916 f81b.sanityn-new.1917 f81b.sanityn-new.1918 f81b.sanityn-new.1919 f81b.sanityn-new.192 f81b.sanityn-new.1920 f81b.sanityn-new.1921 f81b.sanityn-new.1922 f81b.sanityn-new.1923 f81b.sanityn-new.1924 f81b.sanityn-new.1925 f81b.sanityn-new.1926 f81b.sanityn-new.1927 f81b.sanityn-new.1928 f81b.sanityn-new.1929 f81b.sanityn-new.193 f81b.sanityn-new.1930 f81b.sanityn-new.1931 f81b.sanityn-new.1932 f81b.sanityn-new.1933 f81b.sanityn-new.1934 f81b.sanityn-new.1935 f81b.sanityn-new.1936 f81b.sanityn-new.1937 f81b.sanityn-new.1938 f81b.sanityn-new.1939 f81b.sanityn-new.194 f81b.sanityn-new.1940 f81b.sanityn-new.1941 f81b.sanityn-new.1942 f81b.sanityn-new.1943 f81b.sanityn-new.1944 f81b.sanityn-new.1945 f81b.sanityn-new.1946 f81b.sanityn-new.1947 f81b.sanityn-new.1948 f81b.sanityn-new.1949 f81b.sanityn-new.195 f81b.sanityn-new.1950 f81b.sanityn-new.1951 f81b.sanityn-new.1952 f81b.sanityn-new.1953 f81b.sanityn-new.1954 f81b.sanityn-new.1955 f81b.sanityn-new.1956 f81b.sanityn-new.1957 f81b.sanityn-new.1958 f81b.sanityn-new.1959 f81b.sanityn-new.196 f81b.sanityn-new.1960 f81b.sanityn-new.1961 f81b.sanityn-new.1962 f81b.sanityn-new.1963 f81b.sanityn-new.1964 f81b.sanityn-new.1965 f81b.sanityn-new.1966 f81b.sanityn-new.1967 f81b.sanityn-new.1968 f81b.sanityn-new.1969 f81b.sanityn-new.197 f81b.sanityn-new.1970 f81b.sanityn-new.1971 f81b.sanityn-new.1972 f81b.sanityn-new.1973 f81b.sanityn-new.1974 f81b.sanityn-new.1975 f81b.sanityn-new.1976 f81b.sanityn-new.1977 f81b.sanityn-new.1978 f81b.sanityn-new.1979 f81b.sanityn-new.198 f81b.sanityn-new.1980 f81b.sanityn-new.1981 f81b.sanityn-new.1982 f81b.sanityn-new.1983 f81b.sanityn-new.1984 f81b.sanityn-new.1985 f81b.sanityn-new.1986 f81b.sanityn-new.1987 f81b.sanityn-new.1988 f81b.sanityn-new.1989 f81b.sanityn-new.199 f81b.sanityn-new.1990 f81b.sanityn-new.1991 f81b.sanityn-new.1992 f81b.sanityn-new.1993 f81b.sanityn-new.1994 f81b.sanityn-new.1995 f81b.sanityn-new.1996 f81b.sanityn-new.1997 f81b.sanityn-new.1998 f81b.sanityn-new.1999 f81b.sanityn-new.2 f81b.sanityn-new.20 f81b.sanityn-new.200 f81b.sanityn-new.2000 f81b.sanityn-new.2001 f81b.sanityn-new.2002 f81b.sanityn-new.2003 f81b.sanityn-new.2004 f81b.sanityn-new.2005 f81b.sanityn-new.2006 f81b.sanityn-new.2007 f81b.sanityn-new.2008 f81b.sanityn-new.2009 f81b.sanityn-new.201 f81b.sanityn-new.2010 f81b.sanityn-new.2011 f81b.sanityn-new.2012 f81b.sanityn-new.2013 f81b.sanityn-new.2014 f81b.sanityn-new.2015 f81b.sanityn-new.2016 f81b.sanityn-new.2017 f81b.sanityn-new.2018 f81b.sanityn-new.2019 f81b.sanityn-new.202 f81b.sanityn-new.2020 f81b.sanityn-new.2021 f81b.sanityn-new.2022 f81b.sanityn-new.2023 f81b.sanityn-new.2024 f81b.sanityn-new.2025 f81b.sanityn-new.2026 f81b.sanityn-new.2027 f81b.sanityn-new.2028 f81b.sanityn-new.2029 f81b.sanityn-new.203 f81b.sanityn-new.2030 f81b.sanityn-new.2031 f81b.sanityn-new.2032 f81b.sanityn-new.2033 f81b.sanityn-new.2034 f81b.sanityn-new.2035 f81b.sanityn-new.2036 f81b.sanityn-new.2037 f81b.sanityn-new.2038 f81b.sanityn-new.2039 f81b.sanityn-new.204 f81b.sanityn-new.2040 f81b.sanityn-new.2041 f81b.sanityn-new.2042 f81b.sanityn-new.2043 f81b.sanityn-new.2044 f81b.sanityn-new.2045 f81b.sanityn-new.2046 f81b.sanityn-new.2047 f81b.sanityn-new.2048 f81b.sanityn-new.2049 f81b.sanityn-new.205 f81b.sanityn-new.2050 f81b.sanityn-new.2051 f81b.sanityn-new.2052 f81b.sanityn-new.2053 f81b.sanityn-new.2054 f81b.sanityn-new.2055 f81b.sanityn-new.2056 f81b.sanityn-new.2057 f81b.sanityn-new.2058 f81b.sanityn-new.2059 f81b.sanityn-new.206 f81b.sanityn-new.2060 f81b.sanityn-new.2061 f81b.sanityn-new.2062 f81b.sanityn-new.2063 f81b.sanityn-new.2064 f81b.sanityn-new.2065 f81b.sanityn-new.2066 f81b.sanityn-new.2067 f81b.sanityn-new.2068 f81b.sanityn-new.2069 f81b.sanityn-new.207 f81b.sanityn-new.2070 f81b.sanityn-new.2071 f81b.sanityn-new.2072 f81b.sanityn-new.2073 f81b.sanityn-new.2074 f81b.sanityn-new.2075 f81b.sanityn-new.2076 f81b.sanityn-new.2077 f81b.sanityn-new.2078 f81b.sanityn-new.2079 f81b.sanityn-new.208 f81b.sanityn-new.2080 f81b.sanityn-new.2081 f81b.sanityn-new.2082 f81b.sanityn-new.2083 f81b.sanityn-new.2084 f81b.sanityn-new.2085 f81b.sanityn-new.2086 f81b.sanityn-new.2087 f81b.sanityn-new.2088 f81b.sanityn-new.2089 f81b.sanityn-new.209 f81b.sanityn-new.2090 f81b.sanityn-new.2091 f81b.sanityn-new.2092 f81b.sanityn-new.2093 f81b.sanityn-new.2094 f81b.sanityn-new.2095 f81b.sanityn-new.2096 f81b.sanityn-new.2097 f81b.sanityn-new.2098 f81b.sanityn-new.2099 f81b.sanityn-new.21 f81b.sanityn-new.210 f81b.sanityn-new.2100 f81b.sanityn-new.2101 f81b.sanityn-new.2102 f81b.sanityn-new.2103 f81b.sanityn-new.2104 f81b.sanityn-new.2105 f81b.sanityn-new.2106 f81b.sanityn-new.2107 f81b.sanityn-new.2108 f81b.sanityn-new.2109 f81b.sanityn-new.211 f81b.sanityn-new.2110 f81b.sanityn-new.2111 f81b.sanityn-new.2112 f81b.sanityn-new.2113 f81b.sanityn-new.2114 f81b.sanityn-new.2115 f81b.sanityn-new.2116 f81b.sanityn-new.2117 f81b.sanityn-new.2118 f81b.sanityn-new.2119 f81b.sanityn-new.212 f81b.sanityn-new.2120 f81b.sanityn-new.2121 f81b.sanityn-new.2122 f81b.sanityn-new.2123 f81b.sanityn-new.2124 f81b.sanityn-new.2125 f81b.sanityn-new.2126 f81b.sanityn-new.2127 f81b.sanityn-new.2128 f81b.sanityn-new.2129 f81b.sanityn-new.213 f81b.sanityn-new.2130 f81b.sanityn-new.2131 f81b.sanityn-new.2132 f81b.sanityn-new.2133 f81b.sanityn-new.2134 f81b.sanityn-new.2135 f81b.sanityn-new.2136 f81b.sanityn-new.2137 f81b.sanityn-new.2138 f81b.sanityn-new.2139 f81b.sanityn-new.214 f81b.sanityn-new.2140 f81b.sanityn-new.2141 f81b.sanityn-new.2142 f81b.sanityn-new.2143 f81b.sanityn-new.2144 f81b.sanityn-new.2145 f81b.sanityn-new.2146 f81b.sanityn-new.2147 f81b.sanityn-new.2148 f81b.sanityn-new.2149 f81b.sanityn-new.215 f81b.sanityn-new.2150 f81b.sanityn-new.2151 f81b.sanityn-new.2152 f81b.sanityn-new.2153 f81b.sanityn-new.2154 f81b.sanityn-new.2155 f81b.sanityn-new.2156 f81b.sanityn-new.2157 f81b.sanityn-new.2158 f81b.sanityn-new.2159 f81b.sanityn-new.216 f81b.sanityn-new.2160 f81b.sanityn-new.2161 f81b.sanityn-new.2162 f81b.sanityn-new.2163 f81b.sanityn-new.2164 f81b.sanityn-new.2165 f81b.sanityn-new.2166 f81b.sanityn-new.2167 f81b.sanityn-new.2168 f81b.sanityn-new.2169 f81b.sanityn-new.217 f81b.sanityn-new.2170 f81b.sanityn-new.2171 f81b.sanityn-new.2172 f81b.sanityn-new.2173 f81b.sanityn-new.2174 f81b.sanityn-new.2175 f81b.sanityn-new.2176 f81b.sanityn-new.2177 f81b.sanityn-new.2178 f81b.sanityn-new.2179 f81b.sanityn-new.218 f81b.sanityn-new.2180 f81b.sanityn-new.2181 f81b.sanityn-new.2182 f81b.sanityn-new.2183 f81b.sanityn-new.2184 f81b.sanityn-new.2185 f81b.sanityn-new.2186 f81b.sanityn-new.2187 f81b.sanityn-new.2188 f81b.sanityn-new.2189 f81b.sanityn-new.219 f81b.sanityn-new.2190 f81b.sanityn-new.2191 f81b.sanityn-new.2192 f81b.sanityn-new.2193 f81b.sanityn-new.2194 f81b.sanityn-new.2195 f81b.sanityn-new.2196 f81b.sanityn-new.2197 f81b.sanityn-new.2198 f81b.sanityn-new.2199 f81b.sanityn-new.22 f81b.sanityn-new.220 f81b.sanityn-new.2200 f81b.sanityn-new.2201 f81b.sanityn-new.2202 f81b.sanityn-new.2203 f81b.sanityn-new.2204 f81b.sanityn-new.2205 f81b.sanityn-new.2206 f81b.sanityn-new.2207 f81b.sanityn-new.2208 f81b.sanityn-new.2209 f81b.sanityn-new.221 f81b.sanityn-new.2210 f81b.sanityn-new.2211 f81b.sanityn-new.2212 f81b.sanityn-new.2213 f81b.sanityn-new.2214 f81b.sanityn-new.2215 f81b.sanityn-new.2216 f81b.sanityn-new.2217 f81b.sanityn-new.2218 f81b.sanityn-new.2219 f81b.sanityn-new.222 f81b.sanityn-new.2220 f81b.sanityn-new.2221 f81b.sanityn-new.2222 f81b.sanityn-new.2223 f81b.sanityn-new.2224 f81b.sanityn-new.2225 f81b.sanityn-new.2226 f81b.sanityn-new.2227 f81b.sanityn-new.2228 f81b.sanityn-new.2229 f81b.sanityn-new.223 f81b.sanityn-new.2230 f81b.sanityn-new.2231 f81b.sanityn-new.2232 f81b.sanityn-new.2233 f81b.sanityn-new.2234 f81b.sanityn-new.2235 f81b.sanityn-new.2236 f81b.sanityn-new.2237 f81b.sanityn-new.2238 f81b.sanityn-new.2239 f81b.sanityn-new.224 f81b.sanityn-new.2240 f81b.sanityn-new.2241 f81b.sanityn-new.2242 f81b.sanityn-new.2243 f81b.sanityn-new.2244 f81b.sanityn-new.2245 f81b.sanityn-new.2246 f81b.sanityn-new.2247 f81b.sanityn-new.2248 f81b.sanityn-new.2249 f81b.sanityn-new.225 f81b.sanityn-new.2250 f81b.sanityn-new.2251 f81b.sanityn-new.2252 f81b.sanityn-new.2253 f81b.sanityn-new.2254 f81b.sanityn-new.2255 f81b.sanityn-new.2256 f81b.sanityn-new.2257 f81b.sanityn-new.2258 f81b.sanityn-new.2259 f81b.sanityn-new.226 f81b.sanityn-new.2260 f81b.sanityn-new.2261 f81b.sanityn-new.2262 f81b.sanityn-new.2263 f81b.sanityn-new.2264 f81b.sanityn-new.2265 f81b.sanityn-new.2266 f81b.sanityn-new.2267 f81b.sanityn-new.2268 f81b.sanityn-new.2269 f81b.sanityn-new.227 f81b.sanityn-new.2270 f81b.sanityn-new.2271 f81b.sanityn-new.2272 f81b.sanityn-new.2273 f81b.sanityn-new.2274 f81b.sanityn-new.2275 f81b.sanityn-new.2276 f81b.sanityn-new.2277 f81b.sanityn-new.2278 f81b.sanityn-new.2279 f81b.sanityn-new.228 f81b.sanityn-new.2280 f81b.sanityn-new.2281 f81b.sanityn-new.2282 f81b.sanityn-new.2283 f81b.sanityn-new.2284 f81b.sanityn-new.2285 f81b.sanityn-new.2286 f81b.sanityn-new.2287 f81b.sanityn-new.2288 f81b.sanityn-new.2289 f81b.sanityn-new.229 f81b.sanityn-new.2290 f81b.sanityn-new.2291 f81b.sanityn-new.2292 f81b.sanityn-new.2293 f81b.sanityn-new.2294 f81b.sanityn-new.2295 f81b.sanityn-new.2296 f81b.sanityn-new.2297 f81b.sanityn-new.2298 f81b.sanityn-new.2299 f81b.sanityn-new.23 f81b.sanityn-new.230 f81b.sanityn-new.2300 f81b.sanityn-new.2301 f81b.sanityn-new.2302 f81b.sanityn-new.2303 f81b.sanityn-new.2304 f81b.sanityn-new.2305 f81b.sanityn-new.2306 f81b.sanityn-new.2307 f81b.sanityn-new.2308 f81b.sanityn-new.2309 f81b.sanityn-new.231 f81b.sanityn-new.2310 f81b.sanityn-new.2311 f81b.sanityn-new.2312 f81b.sanityn-new.2313 f81b.sanityn-new.2314 f81b.sanityn-new.2315 f81b.sanityn-new.2316 f81b.sanityn-new.2317 f81b.sanityn-new.2318 f81b.sanityn-new.2319 f81b.sanityn-new.232 f81b.sanityn-new.2320 f81b.sanityn-new.2321 f81b.sanityn-new.2322 f81b.sanityn-new.2323 f81b.sanityn-new.2324 f81b.sanityn-new.2325 f81b.sanityn-new.2326 f81b.sanityn-new.2327 f81b.sanityn-new.2328 f81b.sanityn-new.2329 f81b.sanityn-new.233 f81b.sanityn-new.2330 f81b.sanityn-new.2331 f81b.sanityn-new.2332 f81b.sanityn-new.2333 f81b.sanityn-new.2334 f81b.sanityn-new.2335 f81b.sanityn-new.2336 f81b.sanityn-new.2337 f81b.sanityn-new.2338 f81b.sanityn-new.2339 f81b.sanityn-new.234 f81b.sanityn-new.2340 f81b.sanityn-new.2341 f81b.sanityn-new.2342 f81b.sanityn-new.2343 f81b.sanityn-new.2344 f81b.sanityn-new.2345 f81b.sanityn-new.2346 f81b.sanityn-new.2347 f81b.sanityn-new.2348 f81b.sanityn-new.2349 f81b.sanityn-new.235 f81b.sanityn-new.2350 f81b.sanityn-new.2351 f81b.sanityn-new.2352 f81b.sanityn-new.2353 f81b.sanityn-new.2354 f81b.sanityn-new.2355 f81b.sanityn-new.2356 f81b.sanityn-new.2357 f81b.sanityn-new.2358 f81b.sanityn-new.2359 f81b.sanityn-new.236 f81b.sanityn-new.2360 f81b.sanityn-new.2361 f81b.sanityn-new.2362 f81b.sanityn-new.2363 f81b.sanityn-new.2364 f81b.sanityn-new.2365 f81b.sanityn-new.2366 f81b.sanityn-new.2367 f81b.sanityn-new.2368 f81b.sanityn-new.2369 f81b.sanityn-new.237 f81b.sanityn-new.2370 f81b.sanityn-new.2371 f81b.sanityn-new.2372 f81b.sanityn-new.2373 f81b.sanityn-new.2374 f81b.sanityn-new.2375 f81b.sanityn-new.2376 f81b.sanityn-new.2377 f81b.sanityn-new.2378 f81b.sanityn-new.2379 f81b.sanityn-new.238 f81b.sanityn-new.2380 f81b.sanityn-new.2381 f81b.sanityn-new.2382 f81b.sanityn-new.2383 f81b.sanityn-new.2384 f81b.sanityn-new.2385 f81b.sanityn-new.2386 f81b.sanityn-new.2387 f81b.sanityn-new.2388 f81b.sanityn-new.2389 f81b.sanityn-new.239 f81b.sanityn-new.2390 f81b.sanityn-new.2391 f81b.sanityn-new.2392 f81b.sanityn-new.2393 f81b.sanityn-new.2394 f81b.sanityn-new.2395 f81b.sanityn-new.2396 f81b.sanityn-new.2397 f81b.sanityn-new.2398 f81b.sanityn-new.2399 f81b.sanityn-new.24 f81b.sanityn-new.240 f81b.sanityn-new.2400 f81b.sanityn-new.2401 f81b.sanityn-new.2402 f81b.sanityn-new.2403 f81b.sanityn-new.2404 f81b.sanityn-new.2405 f81b.sanityn-new.2406 f81b.sanityn-new.2407 f81b.sanityn-new.2408 f81b.sanityn-new.2409 f81b.sanityn-new.241 f81b.sanityn-new.2410 f81b.sanityn-new.2411 f81b.sanityn-new.2412 f81b.sanityn-new.2413 f81b.sanityn-new.2414 f81b.sanityn-new.2415 f81b.sanityn-new.2416 f81b.sanityn-new.2417 f81b.sanityn-new.2418 f81b.sanityn-new.2419 f81b.sanityn-new.242 f81b.sanityn-new.2420 f81b.sanityn-new.2421 f81b.sanityn-new.2422 f81b.sanityn-new.2423 f81b.sanityn-new.2424 f81b.sanityn-new.2425 f81b.sanityn-new.2426 f81b.sanityn-new.2427 f81b.sanityn-new.2428 f81b.sanityn-new.2429 f81b.sanityn-new.243 f81b.sanityn-new.2430 f81b.sanityn-new.2431 f81b.sanityn-new.2432 f81b.sanityn-new.2433 f81b.sanityn-new.2434 f81b.sanityn-new.2435 f81b.sanityn-new.2436 f81b.sanityn-new.2437 f81b.sanityn-new.2438 f81b.sanityn-new.2439 f81b.sanityn-new.244 f81b.sanityn-new.2440 f81b.sanityn-new.2441 f81b.sanityn-new.2442 f81b.sanityn-new.2443 f81b.sanityn-new.2444 f81b.sanityn-new.2445 f81b.sanityn-new.2446 f81b.sanityn-new.2447 f81b.sanityn-new.2448 f81b.sanityn-new.2449 f81b.sanityn-new.245 f81b.sanityn-new.2450 f81b.sanityn-new.2451 f81b.sanityn-new.2452 f81b.sanityn-new.2453 f81b.sanityn-new.2454 f81b.sanityn-new.2455 f81b.sanityn-new.2456 f81b.sanityn-new.2457 f81b.sanityn-new.2458 f81b.sanityn-new.2459 f81b.sanityn-new.246 f81b.sanityn-new.2460 f81b.sanityn-new.2461 f81b.sanityn-new.2462 f81b.sanityn-new.2463 f81b.sanityn-new.2464 f81b.sanityn-new.2465 f81b.sanityn-new.2466 f81b.sanityn-new.2467 f81b.sanityn-new.2468 f81b.sanityn-new.2469 f81b.sanityn-new.247 f81b.sanityn-new.2470 f81b.sanityn-new.2471 f81b.sanityn-new.2472 f81b.sanityn-new.2473 f81b.sanityn-new.2474 f81b.sanityn-new.2475 f81b.sanityn-new.2476 f81b.sanityn-new.2477 f81b.sanityn-new.2478 f81b.sanityn-new.2479 f81b.sanityn-new.248 f81b.sanityn-new.2480 f81b.sanityn-new.2481 f81b.sanityn-new.2482 f81b.sanityn-new.2483 f81b.sanityn-new.2484 f81b.sanityn-new.2485 f81b.sanityn-new.2486 f81b.sanityn-new.2487 f81b.sanityn-new.2488 f81b.sanityn-new.2489 f81b.sanityn-new.249 f81b.sanityn-new.2490 f81b.sanityn-new.2491 f81b.sanityn-new.2492 f81b.sanityn-new.2493 f81b.sanityn-new.2494 f81b.sanityn-new.2495 f81b.sanityn-new.2496 f81b.sanityn-new.2497 f81b.sanityn-new.2498 f81b.sanityn-new.2499 f81b.sanityn-new.25 f81b.sanityn-new.250 f81b.sanityn-new.251 f81b.sanityn-new.252 f81b.sanityn-new.253 f81b.sanityn-new.254 f81b.sanityn-new.255 f81b.sanityn-new.256 f81b.sanityn-new.257 f81b.sanityn-new.258 f81b.sanityn-new.259 f81b.sanityn-new.26 f81b.sanityn-new.260 f81b.sanityn-new.261 f81b.sanityn-new.262 f81b.sanityn-new.263 f81b.sanityn-new.264 f81b.sanityn-new.265 f81b.sanityn-new.266 f81b.sanityn-new.267 f81b.sanityn-new.268 f81b.sanityn-new.269 f81b.sanityn-new.27 f81b.sanityn-new.270 f81b.sanityn-new.271 f81b.sanityn-new.272 f81b.sanityn-new.273 f81b.sanityn-new.274 f81b.sanityn-new.275 f81b.sanityn-new.276 f81b.sanityn-new.277 f81b.sanityn-new.278 f81b.sanityn-new.279 f81b.sanityn-new.28 f81b.sanityn-new.280 f81b.sanityn-new.281 f81b.sanityn-new.282 f81b.sanityn-new.283 f81b.sanityn-new.284 f81b.sanityn-new.285 f81b.sanityn-new.286 f81b.sanityn-new.287 f81b.sanityn-new.288 f81b.sanityn-new.289 f81b.sanityn-new.29 f81b.sanityn-new.290 f81b.sanityn-new.291 f81b.sanityn-new.292 f81b.sanityn-new.293 f81b.sanityn-new.294 f81b.sanityn-new.295 f81b.sanityn-new.296 f81b.sanityn-new.297 f81b.sanityn-new.298 f81b.sanityn-new.299 f81b.sanityn-new.3 f81b.sanityn-new.30 f81b.sanityn-new.300 f81b.sanityn-new.301 f81b.sanityn-new.302 f81b.sanityn-new.303 f81b.sanityn-new.304 f81b.sanityn-new.305 f81b.sanityn-new.306 f81b.sanityn-new.307 f81b.sanityn-new.308 f81b.sanityn-new.309 f81b.sanityn-new.31 f81b.sanityn-new.310 f81b.sanityn-new.311 f81b.sanityn-new.312 f81b.sanityn-new.313 f81b.sanityn-new.314 f81b.sanityn-new.315 f81b.sanityn-new.316 f81b.sanityn-new.317 f81b.sanityn-new.318 f81b.sanityn-new.319 f81b.sanityn-new.32 f81b.sanityn-new.320 f81b.sanityn-new.321 f81b.sanityn-new.322 f81b.sanityn-new.323 f81b.sanityn-new.324 f81b.sanityn-new.325 f81b.sanityn-new.326 f81b.sanityn-new.327 f81b.sanityn-new.328 f81b.sanityn-new.329 f81b.sanityn-new.33 f81b.sanityn-new.330 f81b.sanityn-new.331 f81b.sanityn-new.332 f81b.sanityn-new.333 f81b.sanityn-new.334 f81b.sanityn-new.335 f81b.sanityn-new.336 f81b.sanityn-new.337 f81b.sanityn-new.338 f81b.sanityn-new.339 f81b.sanityn-new.34 f81b.sanityn-new.340 f81b.sanityn-new.341 f81b.sanityn-new.342 f81b.sanityn-new.343 f81b.sanityn-new.344 f81b.sanityn-new.345 f81b.sanityn-new.346 f81b.sanityn-new.347 f81b.sanityn-new.348 f81b.sanityn-new.349 f81b.sanityn-new.35 f81b.sanityn-new.350 f81b.sanityn-new.351 f81b.sanityn-new.352 f81b.sanityn-new.353 f81b.sanityn-new.354 f81b.sanityn-new.355 f81b.sanityn-new.356 f81b.sanityn-new.357 f81b.sanityn-new.358 f81b.sanityn-new.359 f81b.sanityn-new.36 f81b.sanityn-new.360 f81b.sanityn-new.361 f81b.sanityn-new.362 f81b.sanityn-new.363 f81b.sanityn-new.364 f81b.sanityn-new.365 f81b.sanityn-new.366 f81b.sanityn-new.367 f81b.sanityn-new.368 f81b.sanityn-new.369 f81b.sanityn-new.37 f81b.sanityn-new.370 f81b.sanityn-new.371 f81b.sanityn-new.372 f81b.sanityn-new.373 f81b.sanityn-new.374 f81b.sanityn-new.375 f81b.sanityn-new.376 f81b.sanityn-new.377 f81b.sanityn-new.378 f81b.sanityn-new.379 f81b.sanityn-new.38 f81b.sanityn-new.380 f81b.sanityn-new.381 f81b.sanityn-new.382 f81b.sanityn-new.383 f81b.sanityn-new.384 f81b.sanityn-new.385 f81b.sanityn-new.386 f81b.sanityn-new.387 f81b.sanityn-new.388 f81b.sanityn-new.389 f81b.sanityn-new.39 f81b.sanityn-new.390 f81b.sanityn-new.391 f81b.sanityn-new.392 f81b.sanityn-new.393 f81b.sanityn-new.394 f81b.sanityn-new.395 f81b.sanityn-new.396 f81b.sanityn-new.397 f81b.sanityn-new.398 f81b.sanityn-new.399 f81b.sanityn-new.4 f81b.sanityn-new.40 f81b.sanityn-new.400 f81b.sanityn-new.401 f81b.sanityn-new.402 f81b.sanityn-new.403 f81b.sanityn-new.404 f81b.sanityn-new.405 f81b.sanityn-new.406 f81b.sanityn-new.407 f81b.sanityn-new.408 f81b.sanityn-new.409 f81b.sanityn-new.41 f81b.sanityn-new.410 f81b.sanityn-new.411 f81b.sanityn-new.412 f81b.sanityn-new.413 f81b.sanityn-new.414 f81b.sanityn-new.415 f81b.sanityn-new.416 f81b.sanityn-new.417 f81b.sanityn-new.418 f81b.sanityn-new.419 f81b.sanityn-new.42 f81b.sanityn-new.420 f81b.sanityn-new.421 f81b.sanityn-new.422 f81b.sanityn-new.423 f81b.sanityn-new.424 f81b.sanityn-new.425 f81b.sanityn-new.426 f81b.sanityn-new.427 f81b.sanityn-new.428 f81b.sanityn-new.429 f81b.sanityn-new.43 f81b.sanityn-new.430 f81b.sanityn-new.431 f81b.sanityn-new.432 f81b.sanityn-new.433 f81b.sanityn-new.434 f81b.sanityn-new.435 f81b.sanityn-new.436 f81b.sanityn-new.437 f81b.sanityn-new.438 f81b.sanityn-new.439 f81b.sanityn-new.44 f81b.sanityn-new.440 f81b.sanityn-new.441 f81b.sanityn-new.442 f81b.sanityn-new.443 f81b.sanityn-new.444 f81b.sanityn-new.445 f81b.sanityn-new.446 f81b.sanityn-new.447 f81b.sanityn-new.448 f81b.sanityn-new.449 f81b.sanityn-new.45 f81b.sanityn-new.450 f81b.sanityn-new.451 f81b.sanityn-new.452 f81b.sanityn-new.453 f81b.sanityn-new.454 f81b.sanityn-new.455 f81b.sanityn-new.456 f81b.sanityn-new.457 f81b.sanityn-new.458 f81b.sanityn-new.459 f81b.sanityn-new.46 f81b.sanityn-new.460 f81b.sanityn-new.461 f81b.sanityn-new.462 f81b.sanityn-new.463 f81b.sanityn-new.464 f81b.sanityn-new.465 f81b.sanityn-new.466 f81b.sanityn-new.467 f81b.sanityn-new.468 f81b.sanityn-new.469 f81b.sanityn-new.47 f81b.sanityn-new.470 f81b.sanityn-new.471 f81b.sanityn-new.472 f81b.sanityn-new.473 f81b.sanityn-new.474 f81b.sanityn-new.475 f81b.sanityn-new.476 f81b.sanityn-new.477 f81b.sanityn-new.478 f81b.sanityn-new.479 f81b.sanityn-new.48 f81b.sanityn-new.480 f81b.sanityn-new.481 f81b.sanityn-new.482 f81b.sanityn-new.483 f81b.sanityn-new.484 f81b.sanityn-new.485 f81b.sanityn-new.486 f81b.sanityn-new.487 f81b.sanityn-new.488 f81b.sanityn-new.489 f81b.sanityn-new.49 f81b.sanityn-new.490 f81b.sanityn-new.491 f81b.sanityn-new.492 f81b.sanityn-new.493 f81b.sanityn-new.494 f81b.sanityn-new.495 f81b.sanityn-new.496 f81b.sanityn-new.497 f81b.sanityn-new.498 f81b.sanityn-new.499 f81b.sanityn-new.5 f81b.sanityn-new.50 f81b.sanityn-new.500 f81b.sanityn-new.501 f81b.sanityn-new.502 f81b.sanityn-new.503 f81b.sanityn-new.504 f81b.sanityn-new.505 f81b.sanityn-new.506 f81b.sanityn-new.507 f81b.sanityn-new.508 f81b.sanityn-new.509 f81b.sanityn-new.51 f81b.sanityn-new.510 f81b.sanityn-new.511 f81b.sanityn-new.512 f81b.sanityn-new.513 f81b.sanityn-new.514 f81b.sanityn-new.515 f81b.sanityn-new.516 f81b.sanityn-new.517 f81b.sanityn-new.518 f81b.sanityn-new.519 f81b.sanityn-new.52 f81b.sanityn-new.520 f81b.sanityn-new.521 f81b.sanityn-new.522 f81b.sanityn-new.523 f81b.sanityn-new.524 f81b.sanityn-new.525 f81b.sanityn-new.526 f81b.sanityn-new.527 f81b.sanityn-new.528 f81b.sanityn-new.529 f81b.sanityn-new.53 f81b.sanityn-new.530 f81b.sanityn-new.531 f81b.sanityn-new.532 f81b.sanityn-new.533 f81b.sanityn-new.534 f81b.sanityn-new.535 f81b.sanityn-new.536 f81b.sanityn-new.537 f81b.sanityn-new.538 f81b.sanityn-new.539 f81b.sanityn-new.54 f81b.sanityn-new.540 f81b.sanityn-new.541 f81b.sanityn-new.542 f81b.sanityn-new.543 f81b.sanityn-new.544 f81b.sanityn-new.545 f81b.sanityn-new.546 f81b.sanityn-new.547 f81b.sanityn-new.548 f81b.sanityn-new.549 f81b.sanityn-new.55 f81b.sanityn-new.550 f81b.sanityn-new.551 f81b.sanityn-new.552 f81b.sanityn-new.553 f81b.sanityn-new.554 f81b.sanityn-new.555 f81b.sanityn-new.556 f81b.sanityn-new.557 f81b.sanityn-new.558 f81b.sanityn-new.559 f81b.sanityn-new.56 f81b.sanityn-new.560 f81b.sanityn-new.561 f81b.sanityn-new.562 f81b.sanityn-new.563 f81b.sanityn-new.564 f81b.sanityn-new.565 f81b.sanityn-new.566 f81b.sanityn-new.567 f81b.sanityn-new.568 f81b.sanityn-new.569 f81b.sanityn-new.57 f81b.sanityn-new.570 f81b.sanityn-new.571 f81b.sanityn-new.572 f81b.sanityn-new.573 f81b.sanityn-new.574 f81b.sanityn-new.575 f81b.sanityn-new.576 f81b.sanityn-new.577 f81b.sanityn-new.578 f81b.sanityn-new.579 f81b.sanityn-new.58 f81b.sanityn-new.580 f81b.sanityn-new.581 f81b.sanityn-new.582 f81b.sanityn-new.583 f81b.sanityn-new.584 f81b.sanityn-new.585 f81b.sanityn-new.586 f81b.sanityn-new.587 f81b.sanityn-new.588 f81b.sanityn-new.589 f81b.sanityn-new.59 f81b.sanityn-new.590 f81b.sanityn-new.591 f81b.sanityn-new.592 f81b.sanityn-new.593 f81b.sanityn-new.594 f81b.sanityn-new.595 f81b.sanityn-new.596 f81b.sanityn-new.597 f81b.sanityn-new.598 f81b.sanityn-new.599 f81b.sanityn-new.6 f81b.sanityn-new.60 f81b.sanityn-new.600 f81b.sanityn-new.601 f81b.sanityn-new.602 f81b.sanityn-new.603 f81b.sanityn-new.604 f81b.sanityn-new.605 f81b.sanityn-new.606 f81b.sanityn-new.607 f81b.sanityn-new.608 f81b.sanityn-new.609 f81b.sanityn-new.61 f81b.sanityn-new.610 f81b.sanityn-new.611 f81b.sanityn-new.612 f81b.sanityn-new.613 f81b.sanityn-new.614 f81b.sanityn-new.615 f81b.sanityn-new.616 f81b.sanityn-new.617 f81b.sanityn-new.618 f81b.sanityn-new.619 f81b.sanityn-new.62 f81b.sanityn-new.620 f81b.sanityn-new.621 f81b.sanityn-new.622 f81b.sanityn-new.623 f81b.sanityn-new.624 f81b.sanityn-new.625 f81b.sanityn-new.626 f81b.sanityn-new.627 f81b.sanityn-new.628 f81b.sanityn-new.629 f81b.sanityn-new.63 f81b.sanityn-new.630 f81b.sanityn-new.631 f81b.sanityn-new.632 f81b.sanityn-new.633 f81b.sanityn-new.634 f81b.sanityn-new.635 f81b.sanityn-new.636 f81b.sanityn-new.637 f81b.sanityn-new.638 f81b.sanityn-new.639 f81b.sanityn-new.64 f81b.sanityn-new.640 f81b.sanityn-new.641 f81b.sanityn-new.642 f81b.sanityn-new.643 f81b.sanityn-new.644 f81b.sanityn-new.645 f81b.sanityn-new.646 f81b.sanityn-new.647 f81b.sanityn-new.648 f81b.sanityn-new.649 f81b.sanityn-new.65 f81b.sanityn-new.650 f81b.sanityn-new.651 f81b.sanityn-new.652 f81b.sanityn-new.653 f81b.sanityn-new.654 f81b.sanityn-new.655 f81b.sanityn-new.656 f81b.sanityn-new.657 f81b.sanityn-new.658 f81b.sanityn-new.659 f81b.sanityn-new.66 f81b.sanityn-new.660 f81b.sanityn-new.661 f81b.sanityn-new.662 f81b.sanityn-new.663 f81b.sanityn-new.664 f81b.sanityn-new.665 f81b.sanityn-new.666 f81b.sanityn-new.667 f81b.sanityn-new.668 f81b.sanityn-new.669 f81b.sanityn-new.67 f81b.sanityn-new.670 f81b.sanityn-new.671 f81b.sanityn-new.672 f81b.sanityn-new.673 f81b.sanityn-new.674 f81b.sanityn-new.675 f81b.sanityn-new.676 f81b.sanityn-new.677 f81b.sanityn-new.678 f81b.sanityn-new.679 f81b.sanityn-new.68 f81b.sanityn-new.680 f81b.sanityn-new.681 f81b.sanityn-new.682 f81b.sanityn-new.683 f81b.sanityn-new.684 f81b.sanityn-new.685 f81b.sanityn-new.686 f81b.sanityn-new.687 f81b.sanityn-new.688 f81b.sanityn-new.689 f81b.sanityn-new.69 f81b.sanityn-new.690 f81b.sanityn-new.691 f81b.sanityn-new.692 f81b.sanityn-new.693 f81b.sanityn-new.694 f81b.sanityn-new.695 f81b.sanityn-new.696 f81b.sanityn-new.697 f81b.sanityn-new.698 f81b.sanityn-new.699 f81b.sanityn-new.7 f81b.sanityn-new.70 f81b.sanityn-new.700 f81b.sanityn-new.701 f81b.sanityn-new.702 f81b.sanityn-new.703 f81b.sanityn-new.704 f81b.sanityn-new.705 f81b.sanityn-new.706 f81b.sanityn-new.707 f81b.sanityn-new.708 f81b.sanityn-new.709 f81b.sanityn-new.71 f81b.sanityn-new.710 f81b.sanityn-new.711 f81b.sanityn-new.712 f81b.sanityn-new.713 f81b.sanityn-new.714 f81b.sanityn-new.715 f81b.sanityn-new.716 f81b.sanityn-new.717 f81b.sanityn-new.718 f81b.sanityn-new.719 f81b.sanityn-new.72 f81b.sanityn-new.720 f81b.sanityn-new.721 f81b.sanityn-new.722 f81b.sanityn-new.723 f81b.sanityn-new.724 f81b.sanityn-new.725 f81b.sanityn-new.726 f81b.sanityn-new.727 f81b.sanityn-new.728 f81b.sanityn-new.729 f81b.sanityn-new.73 f81b.sanityn-new.730 f81b.sanityn-new.731 f81b.sanityn-new.732 f81b.sanityn-new.733 f81b.sanityn-new.734 f81b.sanityn-new.735 f81b.sanityn-new.736 f81b.sanityn-new.737 f81b.sanityn-new.738 f81b.sanityn-new.739 f81b.sanityn-new.74 f81b.sanityn-new.740 f81b.sanityn-new.741 f81b.sanityn-new.742 f81b.sanityn-new.743 f81b.sanityn-new.744 f81b.sanityn-new.745 f81b.sanityn-new.746 f81b.sanityn-new.747 f81b.sanityn-new.748 f81b.sanityn-new.749 f81b.sanityn-new.75 f81b.sanityn-new.750 f81b.sanityn-new.751 f81b.sanityn-new.752 f81b.sanityn-new.753 f81b.sanityn-new.754 f81b.sanityn-new.755 f81b.sanityn-new.756 f81b.sanityn-new.757 f81b.sanityn-new.758 f81b.sanityn-new.759 f81b.sanityn-new.76 f81b.sanityn-new.760 f81b.sanityn-new.761 f81b.sanityn-new.762 f81b.sanityn-new.763 f81b.sanityn-new.764 f81b.sanityn-new.765 f81b.sanityn-new.766 f81b.sanityn-new.767 f81b.sanityn-new.768 f81b.sanityn-new.769 f81b.sanityn-new.77 f81b.sanityn-new.770 f81b.sanityn-new.771 f81b.sanityn-new.772 f81b.sanityn-new.773 f81b.sanityn-new.774 f81b.sanityn-new.775 f81b.sanityn-new.776 f81b.sanityn-new.777 f81b.sanityn-new.778 f81b.sanityn-new.779 f81b.sanityn-new.78 f81b.sanityn-new.780 f81b.sanityn-new.781 f81b.sanityn-new.782 f81b.sanityn-new.783 f81b.sanityn-new.784 f81b.sanityn-new.785 f81b.sanityn-new.786 f81b.sanityn-new.787 f81b.sanityn-new.788 f81b.sanityn-new.789 f81b.sanityn-new.79 f81b.sanityn-new.790 f81b.sanityn-new.791 f81b.sanityn-new.792 f81b.sanityn-new.793 f81b.sanityn-new.794 f81b.sanityn-new.795 f81b.sanityn-new.796 f81b.sanityn-new.797 f81b.sanityn-new.798 f81b.sanityn-new.799 f81b.sanityn-new.8 f81b.sanityn-new.80 f81b.sanityn-new.800 f81b.sanityn-new.801 f81b.sanityn-new.802 f81b.sanityn-new.803 f81b.sanityn-new.804 f81b.sanityn-new.805 f81b.sanityn-new.806 f81b.sanityn-new.807 f81b.sanityn-new.808 f81b.sanityn-new.809 f81b.sanityn-new.81 f81b.sanityn-new.810 f81b.sanityn-new.811 f81b.sanityn-new.812 f81b.sanityn-new.813 f81b.sanityn-new.814 f81b.sanityn-new.815 f81b.sanityn-new.816 f81b.sanityn-new.817 f81b.sanityn-new.818 f81b.sanityn-new.819 f81b.sanityn-new.82 f81b.sanityn-new.820 f81b.sanityn-new.821 f81b.sanityn-new.822 f81b.sanityn-new.823 f81b.sanityn-new.824 f81b.sanityn-new.825 f81b.sanityn-new.826 f81b.sanityn-new.827 f81b.sanityn-new.828 f81b.sanityn-new.829 f81b.sanityn-new.83 f81b.sanityn-new.830 f81b.sanityn-new.831 f81b.sanityn-new.832 f81b.sanityn-new.833 f81b.sanityn-new.834 f81b.sanityn-new.835 f81b.sanityn-new.836 f81b.sanityn-new.837 f81b.sanityn-new.838 f81b.sanityn-new.839 f81b.sanityn-new.84 f81b.sanityn-new.840 f81b.sanityn-new.841 f81b.sanityn-new.842 f81b.sanityn-new.843 f81b.sanityn-new.844 f81b.sanityn-new.845 f81b.sanityn-new.846 f81b.sanityn-new.847 f81b.sanityn-new.848 f81b.sanityn-new.849 f81b.sanityn-new.85 f81b.sanityn-new.850 f81b.sanityn-new.851 f81b.sanityn-new.852 f81b.sanityn-new.853 f81b.sanityn-new.854 f81b.sanityn-new.855 f81b.sanityn-new.856 f81b.sanityn-new.857 f81b.sanityn-new.858 f81b.sanityn-new.859 f81b.sanityn-new.86 f81b.sanityn-new.860 f81b.sanityn-new.861 f81b.sanityn-new.862 f81b.sanityn-new.863 f81b.sanityn-new.864 f81b.sanityn-new.865 f81b.sanityn-new.866 f81b.sanityn-new.867 f81b.sanityn-new.868 f81b.sanityn-new.869 f81b.sanityn-new.87 f81b.sanityn-new.870 f81b.sanityn-new.871 f81b.sanityn-new.872 f81b.sanityn-new.873 f81b.sanityn-new.874 f81b.sanityn-new.875 f81b.sanityn-new.876 f81b.sanityn-new.877 f81b.sanityn-new.878 f81b.sanityn-new.879 f81b.sanityn-new.88 f81b.sanityn-new.880 f81b.sanityn-new.881 f81b.sanityn-new.882 f81b.sanityn-new.883 f81b.sanityn-new.884 f81b.sanityn-new.885 f81b.sanityn-new.886 f81b.sanityn-new.887 f81b.sanityn-new.888 f81b.sanityn-new.889 f81b.sanityn-new.89 f81b.sanityn-new.890 f81b.sanityn-new.891 f81b.sanityn-new.892 f81b.sanityn-new.893 f81b.sanityn-new.894 f81b.sanityn-new.895 f81b.sanityn-new.896 f81b.sanityn-new.897 f81b.sanityn-new.898 f81b.sanityn-new.899 f81b.sanityn-new.9 f81b.sanityn-new.90 f81b.sanityn-new.900 f81b.sanityn-new.901 f81b.sanityn-new.902 f81b.sanityn-new.903 f81b.sanityn-new.904 f81b.sanityn-new.905 f81b.sanityn-new.906 f81b.sanityn-new.907 f81b.sanityn-new.908 f81b.sanityn-new.909 f81b.sanityn-new.91 f81b.sanityn-new.910 f81b.sanityn-new.911 f81b.sanityn-new.912 f81b.sanityn-new.913 f81b.sanityn-new.914 f81b.sanityn-new.915 f81b.sanityn-new.916 f81b.sanityn-new.917 f81b.sanityn-new.918 f81b.sanityn-new.919 f81b.sanityn-new.92 f81b.sanityn-new.920 f81b.sanityn-new.921 f81b.sanityn-new.922 f81b.sanityn-new.923 f81b.sanityn-new.924 f81b.sanityn-new.925 f81b.sanityn-new.926 f81b.sanityn-new.927 f81b.sanityn-new.928 f81b.sanityn-new.929 f81b.sanityn-new.93 f81b.sanityn-new.930 f81b.sanityn-new.931 f81b.sanityn-new.932 f81b.sanityn-new.933 f81b.sanityn-new.934 f81b.sanityn-new.935 f81b.sanityn-new.936 f81b.sanityn-new.937 f81b.sanityn-new.938 f81b.sanityn-new.939 f81b.sanityn-new.94 f81b.sanityn-new.940 f81b.sanityn-new.941 f81b.sanityn-new.942 f81b.sanityn-new.943 f81b.sanityn-new.944 f81b.sanityn-new.945 f81b.sanityn-new.946 f81b.sanityn-new.947 f81b.sanityn-new.948 f81b.sanityn-new.949 f81b.sanityn-new.95 f81b.sanityn-new.950 f81b.sanityn-new.951 f81b.sanityn-new.952 f81b.sanityn-new.953 f81b.sanityn-new.954 f81b.sanityn-new.955 f81b.sanityn-new.956 f81b.sanityn-new.957 f81b.sanityn-new.958 f81b.sanityn-new.959 f81b.sanityn-new.96 f81b.sanityn-new.960 f81b.sanityn-new.961 f81b.sanityn-new.962 f81b.sanityn-new.963 f81b.sanityn-new.964 f81b.sanityn-new.965 f81b.sanityn-new.966 f81b.sanityn-new.967 f81b.sanityn-new.968 f81b.sanityn-new.969 f81b.sanityn-new.97 f81b.sanityn-new.970 f81b.sanityn-new.971 f81b.sanityn-new.972 f81b.sanityn-new.973 f81b.sanityn-new.974 f81b.sanityn-new.975 f81b.sanityn-new.976 f81b.sanityn-new.977 f81b.sanityn-new.978 f81b.sanityn-new.979 f81b.sanityn-new.98 f81b.sanityn-new.980 f81b.sanityn-new.981 f81b.sanityn-new.982 f81b.sanityn-new.983 f81b.sanityn-new.984 f81b.sanityn-new.985 f81b.sanityn-new.986 f81b.sanityn-new.987 f81b.sanityn-new.988 f81b.sanityn-new.989 f81b.sanityn-new.99 f81b.sanityn-new.990 f81b.sanityn-new.991 f81b.sanityn-new.992 f81b.sanityn-new.993 f81b.sanityn-new.994 f81b.sanityn-new.995 f81b.sanityn-new.996 f81b.sanityn-new.997 f81b.sanityn-new.998 f81b.sanityn-new.999 ./d90.sanityn: f3 ./d91.sanityn: ./f93.sanityn-1: file1 ./f93.sanityn-2: file2 ~ mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 118o (17s) == sanityn test 118p: Setting notrash flag on symlink file ========================================================== 00:15:52 (1779077752) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain Path '/mnt/lustre/f118p.sanityn.lnk' is a symlink --------------e------- /mnt/lustre/f118p.sanityn file: /mnt/lustre/f118p.sanityn.lnk, flags: 0X40 notrash total 76 drwxrwxrwx 2 root root 4096 May 18 00:15 . drwxrwxrwx 94 root root 69632 May 18 00:15 .. Set flags 0X80000/0X80040 OK: rc=0 file: /mnt/lustre/f118p.sanityn, flags: 0X18FD46FB00080040 notrash mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 118p (8s) == sanityn test 118q: The mode of a directory moved into Trash should not be changed ========================================================== 00:16:00 (1779077760) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain total 76 drwxr-xr-x 2 root root 4096 May 18 00:16 . drwxrwxrwx 95 root root 69632 May 18 00:16 .. -rw-r--r-- 1 root root 0 May 18 00:16 f118q.sanityn total 80 drwxrwxrwx 3 root root 4096 May 18 00:15 . drwxrwxrwx 94 root root 69632 May 18 00:16 .. drwxr-xr-x 2 root root 4096 May 18 00:16 d118q.sanityn mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 118q (9s) == sanityn test 118r: TCU: non-root user access Trash Can and restore files ========================================================== 00:16:08 (1779077768) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d118r.sanityn/f118r.sanityn] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [rm] [-r] [/mnt/lustre/d118r.sanityn/f118r.sanityn] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [ls] [/mnt/lustre/d118r.sanityn/.Trash] f118r.sanityn running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [trash] [unrm] [-v] [/mnt/lustre/d118r.sanityn] Unrm stub file 'f118r.sanityn' under directory '/mnt/lustre/d118r.sanityn' Check if '/mnt/lustre/d118r.sanityn/.Trash/f118r.sanityn' is in Trash Can Set flags 0X80002/0X80000 OK: rc=0 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [mkdir] [/mnt/lustre/d118r.sanityn/d118r.sanityn.1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d118r.sanityn/d118r.sanityn.1/f118r.sanityn.1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [rm] [-r] [/mnt/lustre/d118r.sanityn/d118r.sanityn.1] d118r.sanityn.1 f118r.sanityn.1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [ls] [/mnt/lustre/d118r.sanityn/.Trash] d118r.sanityn.1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [trash] [unrm] [/mnt/lustre/d118r.sanityn] mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 118r (8s) == sanityn test 118s: TCU: The FS flags of a file or directory should be kept ========================================================== 00:16:16 (1779077776) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain -u-----A------e------- /mnt/lustre/d118s.sanityn/.Trash/f118s.sanityn -u-----A-------------- /mnt/lustre/d118s.sanityn/.Trash/d118s.sanityn.0 -u-----A-------------- /mnt/lustre/d118s.sanityn/.Trash/d118s.sanityn.1 mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 118s (8s) == sanityn test 118t: TCU: unrmed directory should keep LUSTRE_PROJHERIT_FL flag ========================================================== 00:16:24 (1779077784) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 118t (8s) == sanityn test 118u: TCU: Space usage of deleted files == 00:16:32 (1779077792) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain Waiting for MDT destroys to complete Max OST size: 3561564 IO size: 3498 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 1.3G 22.7M 1.2G 2% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1.3G 5.7M 1.2G 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3.7G 62.7M 3.4G 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.7G 44.4M 3.4G 2% /mnt/lustre[OST:1] filesystem_summary: 7.3G 107.1M 6.8G 2% /mnt/lustre dd: error writing '/mnt/lustre/d118u.sanityn/f118u.sanityn.0': No space left on device 3460+0 records in 3459+0 records out 3627024384 bytes (3.6 GB, 3.4 GiB) copied, 45.4833 s, 79.7 MB/s dd: error writing '/mnt/lustre/d118u.sanityn/f118u.sanityn.1': No space left on device 3478+0 records in 3477+0 records out 3645898752 bytes (3.6 GB, 3.4 GiB) copied, 45.7845 s, 79.6 MB/s total 7102548 drwxr-xr-x 2 root root 4096 May 18 00:17 . drwxrwxrwx 98 root root 69632 May 18 00:16 .. -rw-r--r-- 1 root root 3627024384 May 18 00:17 f118u.sanityn.0 -rw-r--r-- 1 root root 3645898752 May 18 00:18 f118u.sanityn.1 Filesystem Size Used Avail Use% Mounted on 192.168.201.135@tcp:/lustre 7.4G 6.9G 2.0M 100% /mnt/lustre Use% 100% PCT: 100 Waiting for MDT destroys to complete Trash Can usage via ProjID: 4294967294 Disk quotas for prj (pid 4294967294): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 3560452 0 0 - 1 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-MDT0001 0 - 0 - 1 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 3560452 - 0 - 1 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 total 3542096 drwxr-xr-x 2 root root 4096 May 18 00:18 . drwxrwxrwx 98 root root 69632 May 18 00:16 .. -rw-r--r-- 1 root root 3627024384 May 18 00:17 f118u.sanityn.0 Filesystem Size Used Avail Use% Mounted on 192.168.201.135@tcp:/lustre 7.4G 3.5G 3.4G 51% /mnt/lustre total 3560460 drwxr-xr-x 2 root root 4096 May 18 00:18 . drwxr-xr-x 2 root root 4096 May 18 00:18 .. -rw-r--r-- 1 root root 3645898752 May 18 00:18 f118u.sanityn.1 UUID 1K-blocks Used Available Use% Mounted on lustre-OST0000_UUID 3833116 3606184 836 100% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 3605888 1132 100% /mnt/lustre[OST:1] filesystem_summary: 7666232 7212072 1968 100% /mnt/lustre UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 1.3G 22.7M 1.2G 2% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1.3G 5.7M 1.2G 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3.7G 3.4G 836.0K 100% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.7G 44.4M 3.4G 2% /mnt/lustre[OST:1] filesystem_summary: 7.3G 3.5G 3.4G 51% /mnt/lustre mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 118u (120s) == sanityn test 118v: TCU: Trash Can for encrypted files ========================================================== 00:18:32 (1779077912) /usr/bin/which: no fscrypt in (/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:) SKIP: sanityn test_118v This test needs fscrypt userspace tool SKIP 118v (2s) == sanityn test 118w: TCU: repeated deletion with same name for encrypted files ========================================================== 00:18:34 (1779077914) /usr/bin/which: no fscrypt in (/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:) SKIP: sanityn test_118w This test needs fscrypt userspace tool SKIP 118w (2s) == sanityn test 118x: TCU: virtual .Trash for encrypted striped dir ========================================================== 00:18:36 (1779077916) /usr/bin/which: no fscrypt in (/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:) SKIP: sanityn test_118x This test needs fscrypt userspace tool SKIP 118x (2s) == sanityn test 119a: ltrash_purge: verify dry-run mode and --empty mode ========================================================== 00:18:38 (1779077918) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain touch: cannot touch '/mnt/lustre/d119a.sanityn/f': No space left on device sanityn test_119a: @@@@@@ FAIL: failed to write /mnt/lustre/d119a.sanityn/f Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7318:error() = /home/green/git/lustre-release/lustre/tests/sanityn.sh:10564:test_119a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7694:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7757:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7572:run_test() = /home/green/git/lustre-release/lustre/tests/sanityn.sh:10586:main() Dumping lctl log to /tmp/testlogs//sanityn.test_119a.*.1779077922.log rsync: chown "/tmp/testlogs/.sanityn.test_119a.debug_log.oleg135-server.1779077922.log.QlkWDm" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanityn.test_119a.dmesg.oleg135-server.1779077922.log.EgcyfD" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid FAIL 119a (8s) == sanityn test 119b: ltrash_purge: space-based purging with --auto-ost ========================================================== 00:18:46 (1779077926) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg135-server mds-ost sync done. Waiting for MDT destroys to complete OST0 has highest usage: 100% Increase OST0 space usage by creating 74 MB /mnt/lustre/d119b.sanityn/f1 74+0 records in 74+0 records out 77594624 bytes (78 MB, 74 MiB) copied, 1.15708 s, 67.1 MB/s Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg135-server mds-ost sync done. Stripe /mnt/lustre/d119b.sanityn/f2 across all OSTs f2 Test --auto-ost with high threshold (no OSTs selected) ltrash_purge configuration for /mnt/lustre: - MDT_usage=0.0%, OST_usage=99.0%, max_age=0(s) - scan_interval=60(s), threads=4, del_count=10000 - enable_stats=off, auto_ost=on, auto_mdt=off - mode=dry-run trash projid is 4294967294 Trash can usage on projid 4294967294: blocks 0 inodes 1 MDT0000 stats - blocks=353529, free=347725, avail=312022, files=1024000, ffree=1010507 - usage=1.8%, i_usage=1.3% MDT0001 stats - blocks=353529, free=352067, avail=316364, files=1024000, ffree=1014644 - usage=0.5%, i_usage=0.9% OST0000 stats - blocks=958279, free=56733, avail=209, files=262144, ffree=258202 - usage=100.0%, i_usage=1.5% OST0001 stats - blocks=958279, free=927976, avail=870840, files=262144, ffree=258203 - usage=3.4%, i_usage=1.5% Current space usage in lustre: - Max MDT usage: 1.8%, Max OST usage: 100.0% OST0000 (lustre-OST0000_UUID): usage=100.0%, i_usage=1.5%, max=100.0% OST0001 (lustre-OST0001_UUID): usage=3.4%, i_usage=1.5%, max=3.4% - OST0000 (lustre-OST0000_UUID) with usage 100.0% Auto-selected 1 OST(s) for purging Start scanning and purging at 1779077942: Skip: /mnt/lustre/.ltrash is not a trash directory Add path /mnt/lustre/.ltrash/MDT0000 (depth=0) Add path /mnt/lustre/.ltrash/MDT0001 (depth=0) Add path /mnt/lustre/.ltrash/MDT0001/0x240000405:0xc:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xbd2:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc28:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0001/0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x2000013a1:0x5:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0001/tnm (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc12:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0001/0x240000405:0x41:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0001/0x2400013a1:0x71:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0001/0x240000404:0x5af:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc06:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0001/0x2400013a1:0x6d:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0001/0x2400013a1:0x69:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200001b71:0xf0:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0001/0x2400013a1:0x6b:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0001/0x240000404:0x5b1:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200001b73:0xb61e:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc25:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xbcc:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200001b71:0x84:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200001b71:0x7e:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200001b71:0xea:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200001b71:0x77:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xbec:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc1f:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc0c:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd0:0x657:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc95:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200001b71:0x1:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x2000013a1:0x3ee:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x2000013a1:0x3f3:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x2000013a1:0x401:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/500 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000007:0x1:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200001b71:0xa:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x2000013a1:0x3:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200001b71:0x10bc7:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc33:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc37:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200001b73:0xb60b:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc1c:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc2e:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc39:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc93:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200001b73:0xb60c:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xbe3:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200001b71:0x79:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x2000013a1:0x3f8:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/tnm (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200001b71:0xe5:0x0 (depth=1) Purging object: /mnt/lustre/.ltrash/MDT0000/0x200001b73:0xb61e:0x0/f2 dry-run: would purge /mnt/lustre/.ltrash/MDT0000/0x200001b73:0xb61e:0x0/f2 (size=0) Collected 55 dirs totally Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200001b71:0x79:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x2000013a1:0x3f8:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200001b71:0xe5:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/tnm' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200001b73:0xb60c:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xbe3:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc93:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc39:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc37:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc2e:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc1c:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200001b73:0xb60b:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc33:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200001b71:0x10bc7:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x2000013a1:0x3:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200001b71:0xa:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x2000013a1:0x401:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000007:0x1:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/500' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x2000013a1:0x3f3:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x2000013a1:0x3ee:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd0:0x657:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200001b71:0x1:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc95:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xbec:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc1f:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc0c:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200001b71:0xea:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200001b71:0x7e:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200001b71:0x77:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200001b71:0x84:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xbcc:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc25:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0001/0x240000404:0x5b1:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200001b73:0xb61e:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0001/0x2400013a1:0x6b:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200001b71:0xf0:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0001/0x2400013a1:0x69:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0001/0x2400013a1:0x6d:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc06:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0001/0x240000404:0x5af:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0001/0x2400013a1:0x71:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0001/0x240000405:0x41:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc12:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0001/tnm' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x2000013a1:0x5:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc28:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0001/0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xbd2:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0001/0x240000405:0xc:0x0' Deleted 53 directories, 0 errors in depth 1 Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0001' Deleted 2 directories, 0 errors in depth 0 Totally deleted 55 directories, 0 errors ltrash_purge ended with exit code: 0 sanityn test_119b: @@@@@@ FAIL: Expected 'No OSTs exceed threshold' message not found Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7318:error() = /home/green/git/lustre-release/lustre/tests/sanityn.sh:10654:test_119b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7694:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7757:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7572:run_test() = /home/green/git/lustre-release/lustre/tests/sanityn.sh:10708:main() Dumping lctl log to /tmp/testlogs//sanityn.test_119b.*.1779077943.log rsync: chown "/tmp/testlogs/.sanityn.test_119b.debug_log.oleg135-server.1779077943.log.8t500B" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanityn.test_119b.dmesg.oleg135-server.1779077943.log.4srWZs" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid FAIL 119b (21s) == sanityn test 119c: ltrash_purge: age-based purging ==== 00:19:07 (1779077947) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg135-server mds-ost sync done. Waiting for MDT destroys to complete Create and remove /mnt/lustre/d119c.sanityn/old_f1 and /mnt/lustre/d119c.sanityn/old_f2 Waiting 6 seconds for files becoming old Create and remove /mnt/lustre/d119c.sanityn/new_f3 old_f1 old_f2 new_f3 ltrash_purge configuration for /mnt/lustre: - MDT_usage=0.0%, OST_usage=0.0%, max_age=5(s) - scan_interval=60(s), threads=4, del_count=10000 - enable_stats=off, auto_ost=off, auto_mdt=off - mode=daemon Current space usage in lustre: - Max MDT usage: 1.8%, Max OST usage: 100.0% Start scanning and purging at 1779077963: Waiting 4s for '0' Purged: /mnt/lustre/.ltrash/MDT0000/0x200001b73:0xb622:0x0/old_f2 (size=5 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200001b73:0xb622:0x0/old_f1 (size=5 bytes) Exit due to signal: Terminated exiting: Terminated new_f3 Empty mode is enabled, all the filters are ignored ltrash_purge configuration for /mnt/lustre: - MDT_usage=90.0%, OST_usage=90.0%, max_age=604800(s) - scan_interval=60(s), threads=4, del_count=10000 - enable_stats=on, auto_ost=off, auto_mdt=off - mode=empty Start scanning and purging at 1779077964: Purged: /mnt/lustre/.ltrash/MDT0000/0x200001b73:0xb622:0x0/new_f3 (size=4 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc39:0x0 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc37:0x0 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200001b71:0xea:0x0 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0001/0x240000405:0x41:0x0 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0001/0x240000405:0xc:0x0 (size=4096 bytes) Scan and purge completed successfully at 1779077964: - scanned 114 objs, purged 6 objs, failed 0 objs, freed 20484 bytes - work rate 185.44 objs/sec ltrash_purge ended with exit code: 0 mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 119c (22s) == sanityn test 119d: ltrash_purge: id-based filtering (UID/GID/PROJID) ========================================================== 00:19:29 (1779077969) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg135-client.virtnet /mnt/lustre (opts:) Stopping client oleg135-client.virtnet /mnt/lustre opts: Stopping clients: oleg135-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg135-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg135-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg135-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg135-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg135-server tune2fs 1.47.3-wc2 (11-Nov-2025) tune2fs 1.47.3-wc2 (11-Nov-2025) tune2fs 1.47.3-wc2 (11-Nov-2025) tune2fs 1.47.3-wc2 (11-Nov-2025) sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) debugfs on /sys/kernel/debug type debugfs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg135-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.35,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg135-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.35,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg135-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.35,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/63884/testresults/sanityn-ldiskfs-DNE-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.35,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg135-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg135-server: oleg135-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-MDT0000 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg135-server: oleg135-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-MDT0001 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg135-server: oleg135-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg135-server: oleg135-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Starting client oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre Started clients oleg135-client.virtnet: 192.168.201.135@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 Starting client oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre2 Started clients oleg135-client.virtnet: 192.168.201.135@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffffa0e720790800.idle_timeout=debug osc.lustre-OST0000-osc-ffffa0e736659800.idle_timeout=debug osc.lustre-OST0001-osc-ffffa0e720790800.idle_timeout=debug osc.lustre-OST0001-osc-ffffa0e736659800.idle_timeout=debug disable quota as required mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid Trash Can Type: uid Create and remove files as root Create and remove files as 500 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d119d.sanityn/user_f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [rm] [/mnt/lustre/d119d.sanityn/user_f1] Create and remove files with project ID Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg135-server mds-ost sync done. Waiting for MDT destroys to complete root_f1 proj_f1 proj_f2 Testing UID-based filtering for UID=500 ltrash_purge configuration for /mnt/lustre: - MDT_usage=0.0%, OST_usage=0.0%, max_age=0(s) - scan_interval=60(s), threads=4, del_count=10000 - enable_stats=off, auto_ost=off, auto_mdt=off - mode=daemon trash projid is 4294967294 Trash can usage on projid 4294967294: blocks 4096 inodes 4 MDT0000 stats - blocks=353529, free=347748, avail=312045, files=1024000, ffree=1010506 - usage=1.8%, i_usage=1.3% MDT0001 stats - blocks=353529, free=352098, avail=316395, files=1024000, ffree=1014652 - usage=0.5%, i_usage=0.9% OST0000 stats - blocks=958279, free=56732, avail=208, files=262144, ffree=258282 - usage=100.0%, i_usage=1.5% Waiting 90s for '0' OST0001 stats - blocks=958279, free=927970, avail=871446, files=262144, ffree=258217 - usage=3.4%, i_usage=1.5% Current space usage in lustre: - Max MDT usage: 1.8%, Max OST usage: 100.0% Start scanning and purging at 1779078075: Skip: /mnt/lustre/.ltrash is not a trash directory Add path /mnt/lustre/.ltrash/MDT0000 (depth=0) Add path /mnt/lustre/.ltrash/MDT0001 (depth=0) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xbd2:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc28:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x2000013a1:0x5:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc12:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc06:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200001b71:0xf0:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200001b73:0xb61e:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc25:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200001b73:0xb622:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xbcc:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200001b71:0x84:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200001b71:0x7e:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200001b71:0x77:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xbec:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc1f:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc0c:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd0:0x657:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc95:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200001b71:0x1:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x2000013a1:0x3ee:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x2000013a1:0x3f3:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x2000013a1:0x401:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/500 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000007:0x1:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200001b71:0xa:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x2000013a1:0x3:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200001b71:0x10bc7:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc33:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200001b73:0xb60b:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc1c:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc2e:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc93:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200001b73:0xb60c:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xbe3:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200001b71:0x79:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x2000013a1:0x3f8:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/tnm (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200001b71:0xe5:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0001/0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0001/tnm (depth=1) Add path /mnt/lustre/.ltrash/MDT0001/0x2400013a1:0x71:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0001/0x240000404:0x5af:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0001/0x2400013a1:0x6d:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0001/0x2400013a1:0x69:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0001/0x2400013a1:0x6b:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0001/0x240000404:0x5b1:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0/0x200002341:0x1:0x0 (depth=2) Add path /mnt/lustre/.ltrash/MDT0000/500/0x200002341:0x1:0x0 (depth=2) Purging object: /mnt/lustre/.ltrash/MDT0000/500/0x200002341:0x1:0x0/user_f1 Purged: /mnt/lustre/.ltrash/MDT0000/500/0x200002341:0x1:0x0/user_f1 (size=0 bytes) Collected 53 dirs totally Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0/0x200002341:0x1:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/500/0x200002341:0x1:0x0' Deleted 2 directories, 0 errors in depth 2 Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0001/0x240000404:0x5b1:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0001/0x2400013a1:0x6b:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0001/0x2400013a1:0x6d:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0001/0x2400013a1:0x69:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0001/0x240000404:0x5af:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0001/tnm' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0001/0x2400013a1:0x71:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0001/0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200001b71:0xe5:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/tnm' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200001b71:0x79:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x2000013a1:0x3f8:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xbe3:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200001b73:0xb60c:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc93:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc2e:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc1c:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200001b73:0xb60b:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc33:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x2000013a1:0x3:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200001b71:0x10bc7:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200001b71:0xa:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000007:0x1:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/500' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x2000013a1:0x401:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x2000013a1:0x3f3:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x2000013a1:0x3ee:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200001b71:0x1:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc95:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd0:0x657:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc0c:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xbec:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200001b71:0x77:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200001b71:0x7e:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc1f:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200001b71:0x84:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xbcc:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200001b73:0xb622:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc25:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200001b71:0xf0:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200001b73:0xb61e:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc06:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc12:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x2000013a1:0x5:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc28:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xbd2:0x0' Deleted 49 directories, 0 errors in depth 1 Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0001' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000' Deleted 2 directories, 0 errors in depth 0 Totally deleted 53 directories, 0 errors Exit due to signal: Terminated exiting: Terminated root_f1 proj_f1 proj_f2 Testing GID-based filtering for GID=500 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d119d.sanityn/user_f2] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [rm] [/mnt/lustre/d119d.sanityn/user_f2] ltrash_purge configuration for /mnt/lustre: - MDT_usage=0.0%, OST_usage=0.0%, max_age=0(s) - scan_interval=60(s), threads=4, del_count=10000 - enable_stats=off, auto_ost=off, auto_mdt=off - mode=daemon Waiting 90s for '0' Current space usage in lustre: - Max MDT usage: 1.8%, Max OST usage: 100.0% Start scanning and purging at 1779078076: Purged: /mnt/lustre/.ltrash/MDT0000/500/0x200002341:0x1:0x0/user_f2 (size=0 bytes) Exit due to signal: Terminated exiting: Terminated root_f1 proj_f1 Testing PROJID-based filtering for PROJID=100 ltrash_purge configuration for /mnt/lustre: - MDT_usage=0.0%, OST_usage=0.0%, max_age=0(s) - scan_interval=60(s), threads=4, del_count=10000 - enable_stats=off, auto_ost=off, auto_mdt=off - mode=daemon Current space usage in lustre: - Max MDT usage: 1.8%, Max OST usage: 100.0% Start scanning and purging at 1779078077: Waiting 90s for '0' Purged: /mnt/lustre/.ltrash/MDT0000/0/0x200002341:0x1:0x0/proj_f1 (size=0 bytes) Exit due to signal: Terminated exiting: Terminated proj_f2 root_f1 Empty mode is enabled, all the filters are ignored ltrash_purge configuration for /mnt/lustre: - MDT_usage=90.0%, OST_usage=90.0%, max_age=604800(s) - scan_interval=60(s), threads=4, del_count=10000 - enable_stats=off, auto_ost=off, auto_mdt=off - mode=empty Start scanning and purging at 1779078079: Purged: /mnt/lustre/.ltrash/MDT0000/0/0x200002341:0x1:0x0/root_f1 (size=6 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0/0x200002341:0x1:0x0/proj_f2 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/500/0x200002341:0x1:0x0 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0/0x200002341:0x1:0x0 (size=4096 bytes) ltrash_purge ended with exit code: 0 mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 119d (114s) == sanityn test 119e: ltrash_purge: verify --conf option ========================================================== 00:21:23 (1779078083) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg135-server mds-ost sync done. Waiting for MDT destroys to complete f Verify actual purge with config file ltrash_purge configuration for /mnt/lustre: - MDT_usage=0.1%, OST_usage=0.0%, max_age=0(s) - scan_interval=60(s), threads=4, del_count=10000 - enable_stats=off, auto_ost=off, auto_mdt=off - mode=daemon Current space usage in lustre: - Max MDT usage: 1.8%, Max OST usage: 100.0% Start scanning and purging at 1779078095: Waiting 90s for '0' Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0x7:0x0/f (size=5 bytes) Exit due to signal: Terminated exiting: Terminated mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 119e (17s) == sanityn test 119f: Run ltrash_purge on subdir mount === 00:21:40 (1779078100) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre/d119f.sanityn /mnt/lustre_d119f.sanityn f119f.sanityn Empty mode is enabled, all the filters are ignored ltrash_purge configuration for /mnt/lustre_d119f.sanityn: - MDT_usage=90.0%, OST_usage=90.0%, max_age=604800(s) - scan_interval=60(s), threads=4, del_count=10000 - enable_stats=on, auto_ost=off, auto_mdt=off - mode=empty Start scanning and purging at 1779078104: Purged: /mnt/lustre_d119f.sanityn/.ltrash/MDT0000/0x200002341:0x9:0x0/f119f.sanityn (size=0 bytes) Scan and purge completed successfully at 1779078105: - scanned 108 objs, purged 1 objs, failed 0 objs, freed 0 bytes - work rate 189.80 objs/sec ltrash_purge ended with exit code: 0 192.168.201.135@tcp:/lustre/d119f.sanityn /mnt/lustre_d119f.sanityn lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg135-client.virtnet /mnt/lustre_d119f.sanityn (opts:) mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 119f (37s) == sanityn test 119g: ltrash_purge: parallel directory deletion ========================================================== 00:22:18 (1779078138) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain Empty mode is enabled, all the filters are ignored ltrash_purge configuration for /mnt/lustre: - MDT_usage=90.0%, OST_usage=90.0%, max_age=604800(s) - scan_interval=60(s), threads=4, del_count=10000 - enable_stats=off, auto_ost=off, auto_mdt=off - mode=empty Start scanning and purging at 1779078141: ltrash_purge ended with exit code: 0 Creating random directory tree: depth=5 Level 1: created 2 dirs Level 2: created 4 dirs Level 3: created 7 dirs Level 4: created 15 dirs Level 5: created 33 dirs Created 61 dirs and 93 files across 6 levels | | |-- d119g.sanityn | | | |-- d1_1 | | | | |-- f_2 | | | | |-- f_1 | | | | |-- d2_0 | | | | | |-- f_2 | | | | | |-- f_1 | | | | | |-- d3_0 | | | | | | |-- f_2 | | | | | | |-- f_1 | | | | | | |-- f_0 | | | | | | |-- d4_0 | | | | | | | |-- f_2 | | | | | | | |-- f_1 | | | | | | | |-- d5_1 | | | | | | | | |-- f_2 | | | | | | | | |-- f_1 | | | | | | | | |-- f_0 | | | | | | | |-- d5_2 | | | | | | | | |-- f_0 | | | | | | | |-- f_0 | | | | | | | |-- d5_0 | | | | | | | | |-- f_2 | | | | | | | | |-- f_1 | | | | | | | | |-- f_0 | | | | | |-- f_0 | | | | | |-- d3_1 | | | | | | |-- d4_1 | | | | | | | |-- d5_1 | | | | | | | | |-- f_2 | | | | | | | | |-- f_1 | | | | | | | | |-- f_0 | | | | | | | |-- d5_0 | | | | | | | | |-- f_1 | | | | | | | | |-- f_0 | | | | | | |-- d4_0 | | | | | | | |-- f_0 | | | | | | | |-- d5_0 | | | | | | | | |-- f_2 | | | | | | | | |-- f_1 | | | | | | | | |-- f_0 | | | | | | |-- d4_2 | | | | | | | |-- d5_1 | | | | | | | | |-- f_1 | | | | | | | | |-- f_0 | | | | | | | |-- d5_0 | | | | | | | | |-- f_0 | | | | |-- f_0 | | | |-- d1_0 | | | | |-- d2_2 | | | | | |-- d3_0 | | | | | | |-- d4_1 | | | | | | | |-- f_2 | | | | | | | |-- f_1 | | | | | | | |-- d5_1 | | | | | | | |-- d5_2 | | | | | | | | |-- f_0 | | | | | | | |-- f_0 | | | | | | | |-- d5_0 | | | | | | | | |-- f_1 | | | | | | | | |-- f_0 | | | | | | |-- f_0 | | | | | | |-- d4_0 | | | | | | | |-- d5_1 | | | | | | | | |-- f_1 | | | | | | | | |-- f_0 | | | | | | | |-- d5_2 | | | | | | | |-- f_0 | | | | | | | |-- d5_0 | | | | | | | | |-- f_1 | | | | | | | | |-- f_0 | | | | | | |-- d4_2 | | | | | | | |-- f_1 | | | | | | | |-- f_0 | | | | | | | |-- d5_0 | | | | | | | | |-- f_2 | | | | | | | | |-- f_1 | | | | | | | | |-- f_0 | | | | | |-- f_0 | | | | |-- d2_0 | | | | | |-- f_1 | | | | | |-- d3_0 | | | | | | |-- f_1 | | | | | | |-- d4_1 | | | | | | | |-- f_0 | | | | | | | |-- d5_0 | | | | | | | | |-- f_0 | | | | | | |-- f_0 | | | | | | |-- d4_0 | | | | | | | |-- d5_1 | | | | | | | | |-- f_1 | | | | | | | | |-- f_0 | | | | | | | |-- d5_2 | | | | | | | | |-- f_1 | | | | | | | | |-- f_0 | | | | | | | |-- d5_0 | | | | | | | | |-- f_1 | | | | | | | | |-- f_0 | | | | | |-- f_0 | | | | | |-- d3_1 | | | | | | |-- f_1 | | | | | | |-- f_0 | | | | | | |-- d4_0 | | | | | | | |-- d5_1 | | | | | | | |-- d5_2 | | | | | | | | |-- f_2 | | | | | | | | |-- f_1 | | | | | | | | |-- f_0 | | | | | | | |-- d5_0 | | | | |-- d2_1 | | | | | |-- d3_0 | | | | | | |-- f_2 | | | | | | |-- f_1 | | | | | | |-- d4_1 | | | | | | | |-- d5_1 | | | | | | | |-- d5_0 | | | | | | | | |-- f_2 | | | | | | | | |-- f_1 | | | | | | | | |-- f_0 | | | | | | |-- f_0 | | | | | | |-- d4_0 | | | | | | | |-- d5_1 | | | | | | | |-- d5_0 | | | | | | | | |-- f_0 | | | | | | |-- d4_2 | | | | | | | |-- d5_1 | | | | | | | | |-- f_2 | | | | | | | | |-- f_1 | | | | | | | | |-- f_0 | | | | | | | |-- d5_2 | | | | | | | | |-- f_1 | | | | | | | | |-- f_0 | | | | | | | |-- f_0 | | | | | | | |-- d5_0 | | | | | | | | |-- f_2 | | | | | | | | |-- f_1 | | | | | | | | |-- f_0 | | | | | |-- f_0 | | | | | |-- d3_1 | | | | | | |-- d4_1 | | | | | | | |-- d5_1 | | | | | | | | |-- f_2 | | | | | | | | |-- f_1 | | | | | | | | |-- f_0 | | | | | | | |-- d5_0 | | | | | | |-- f_0 | | | | | | |-- d4_0 | | | | | | | |-- f_1 | | | | | | | |-- d5_1 | | | | | | | | |-- f_2 | | | | | | | | |-- f_1 | | | | | | | | |-- f_0 | | | | | | | |-- f_0 | | | | | | | |-- d5_0 | | | | | | | | |-- f_0 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg135-server mds-ost sync done. Waiting for MDT destroys to complete Empty mode is enabled, all the filters are ignored ltrash_purge configuration for /mnt/lustre: - MDT_usage=90.0%, OST_usage=90.0%, max_age=604800(s) - scan_interval=60(s), threads=4, del_count=10000 - enable_stats=on, auto_ost=off, auto_mdt=off - mode=empty Start scanning and purging at 1779078162: Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/f_2 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/f_1 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_2/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_0/f_1 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_0/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_2/d3_0/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0/f_2 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_0/f_2 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_0/d3_0/f_1 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0/f_1 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_0/f_1 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_0/d3_0/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_0/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_1/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_0/d3_1/f_1 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_2/d3_0/d4_0/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_2/d3_0/d4_1/f_2 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_0/d3_1/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_2/d3_0/d4_2/f_1 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_2/d3_0/d4_1/f_1 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_2/d3_0/d4_2/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_2/d3_0/d4_1/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_0/d3_0/d4_1/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0/d3_0/f_2 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0/d3_0/f_1 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0/d3_0/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_0/d4_2/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_1/d4_0/f_1 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_2/d3_0/d4_0/d5_1/f_1 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_1/d4_0/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_2/d3_0/d4_0/d5_0/f_1 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_2/d3_0/d4_0/d5_1/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_2/d3_0/d4_0/d5_0/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_2/d3_0/d4_1/d5_2/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_2/d3_0/d4_2/d5_0/f_2 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_2/d3_0/d4_1/d5_0/f_1 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_0/d3_0/d4_1/d5_0/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_2/d3_0/d4_2/d5_0/f_1 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_2/d3_0/d4_1/d5_0/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_2/d3_0/d4_2/d5_0/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_0/d4_1/d5_0/f_2 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0/d3_1/d4_0/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_0/d4_1/d5_0/f_1 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_0/d4_1/d5_0/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_1/d4_1/d5_1/f_2 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_0/d3_0/d4_0/d5_1/f_1 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_1/d4_1/d5_1/f_1 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_0/d3_0/d4_0/d5_1/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_1/d4_1/d5_1/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_0/d3_0/d4_0/d5_2/f_1 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_0/d3_0/d4_0/d5_0/f_1 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_0/d3_0/d4_0/d5_0/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_0/d3_0/d4_0/d5_2/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0/d3_0/d4_0/f_2 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_0/d4_2/d5_1/f_2 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0/d3_0/d4_0/f_1 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_0/d4_2/d5_2/f_1 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_0/d4_2/d5_1/f_1 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0/d3_0/d4_0/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_0/d4_2/d5_1/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_0/d4_2/d5_2/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_0/d4_0/d5_0/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_0/d4_2/d5_0/f_2 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_1/d4_0/d5_1/f_2 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_0/d4_2/d5_0/f_1 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_1/d4_0/d5_0/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_0/d4_2/d5_0/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_1/d4_0/d5_1/f_1 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_0/d3_1/d4_0/d5_2/f_2 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_1/d4_0/d5_1/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_0/d3_1/d4_0/d5_2/f_1 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0/d3_1/d4_1/d5_1/f_2 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_0/d3_1/d4_0/d5_2/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0/d3_1/d4_2/d5_1/f_1 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0/d3_1/d4_1/d5_1/f_1 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0/d3_1/d4_1/d5_0/f_1 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0/d3_1/d4_1/d5_1/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0/d3_1/d4_2/d5_1/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0/d3_1/d4_1/d5_0/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0/d3_1/d4_2/d5_0/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0/d3_1/d4_0/d5_0/f_2 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0/d3_0/d4_0/d5_1/f_2 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0/d3_0/d4_0/d5_2/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0/d3_1/d4_0/d5_0/f_1 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0/d3_0/d4_0/d5_0/f_2 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0/d3_0/d4_0/d5_1/f_1 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0/d3_1/d4_0/d5_0/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0/d3_0/d4_0/d5_1/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0/d3_0/d4_0/d5_0/f_1 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0/d3_0/d4_0/d5_0/f_0 (size=0 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0/d3_0/d4_0/d5_0 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0/d3_1/d4_0/d5_0 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0/d3_0/d4_0/d5_2 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0/d3_0/d4_0/d5_1 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0/d3_1/d4_1/d5_0 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0/d3_1/d4_2/d5_0 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0/d3_1/d4_2/d5_1 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0/d3_1/d4_1/d5_1 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_0/d3_1/d4_0/d5_0 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_0/d3_1/d4_0/d5_1 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_0/d3_1/d4_0/d5_2 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_1/d4_0/d5_0 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_1/d4_0/d5_1 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_0/d4_2/d5_0 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_0/d4_0/d5_0 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_0/d4_2/d5_2 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_0/d4_0/d5_1 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_0/d4_2/d5_1 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_0/d3_0/d4_0/d5_0 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_0/d3_0/d4_0/d5_2 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_1/d4_1/d5_0 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_0/d3_0/d4_0/d5_1 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_1/d4_1/d5_1 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_0/d4_1/d5_0 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_0/d4_1/d5_1 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_2/d3_0/d4_1/d5_0 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_0/d3_0/d4_1/d5_0 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_2/d3_0/d4_2/d5_0 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_2/d3_0/d4_0/d5_2 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_2/d3_0/d4_1/d5_2 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_2/d3_0/d4_0/d5_0 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_2/d3_0/d4_1/d5_1 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_2/d3_0/d4_0/d5_1 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0/d3_0/d4_0 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0/d3_1/d4_2 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0/d3_1/d4_1 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0/d3_1/d4_0 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_0/d3_1/d4_0 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_1/d4_0 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_0/d4_2 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_0/d4_0 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_0/d3_0/d4_0 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_1/d4_1 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_0/d4_1 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_0/d3_0/d4_1 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_2/d3_0/d4_2 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_2/d3_0/d4_0 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_2/d3_0/d4_1 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_1 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_0/d3_1 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0/d3_1 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0/d3_0 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_0/d3_0 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_2/d3_0 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1/d3_0 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1/d2_0 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_1 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_2 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0/d2_0 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_0 (size=4096 bytes) Purged: /mnt/lustre/.ltrash/MDT0000/0x200002341:0xa:0x0/d1_1 (size=4096 bytes) Scan and purge completed successfully at 1779078163: - scanned 324 objs, purged 154 objs, failed 0 objs, freed 249856 bytes - work rate 458.67 objs/sec ltrash_purge ended with exit code: 0 | | | | |-- 0x200002341:0xa:0x0 mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid PASS 119g (30s) == sanityn test 200: service remains healthy while able to process request ========================================================== 00:22:48 (1779078168) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=3 striped dir -i0 -c2 -H crush2 /mnt/lustre/d200.sanityn fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0945 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 52.5894 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 84.3424 s, 0.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 116.095 s, 0.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 148.825 s, 0.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 180.603 s, 0.0 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg135-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffffa0e720790800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa0e720790800.ost_server_uuid in IDLE state after 0 sec PASS 200 (190s) cleanup: ====================================================== == sanityn test complete, duration 10796 sec ============= 00:25:59 (1779078359) sanityn: FAIL: test_117yb /etc and /mnt/lustre/d117yb.sanityn/etc diff sanityn: FAIL: test_118Aa 0x200001b71:0x78:0x0 should be removed from PENDING sanityn: FAIL: test_118Ab 0x200001b71:0x7a:0x0 should be removed/cleanup from PENDING sanityn: FAIL: test_119a failed to write /mnt/lustre/d119a.sanityn/f sanityn: FAIL: test_119b Expected 'No OSTs exceed threshold' message not found === sanityn: start cleanup 00:26:00 (1779078360) === Stopping clients: oleg135-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg135-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 00:28:14 (1779078494) ===