-----============= acceptance-small: sanityn ============----- Sat May 9 13:19:02 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg435-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a 102 skipping tests SLOW=no: 33a === sanityn: start setup 13:19:17 (1778347157) === Starting client oleg435-client.virtnet: -o user_xattr,flock 192.168.204.135@tcp:/lustre /mnt/lustre2 Mount client oleg435-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.204.135@tcp:/lustre /mnt/lustre2 Started clients oleg435-client.virtnet: 192.168.204.135@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg435-client.virtnet: executing check_config_client /mnt/lustre oleg435-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg435-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9e53829a2000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9e5383546800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e53829a2000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e5383546800.idle_timeout=debug disable quota as required oleg435-server: oleg435-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 13:19:36 (1778347176) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149928 s, 69.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7598] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 13:19:38 (1778347178) PASS 0 (5s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 13:19:43 (1778347183) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (4s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 13:19:47 (1778347187) -rw-r--r-- 1 root root 0 May 9 13:19 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (4s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 13:19:51 (1778347191) -rw-r--r-- 1 root root 0 May 9 13:19 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (5s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 13:19:56 (1778347196) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (4s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 13:20:01 (1778347201) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (4s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 13:20:05 (1778347205) total 1 -rwxrwxrwx 1 root root 0 May 9 13:19 f2a -rwxrwxrwx 1 root root 0 May 9 13:19 f2b total 1 -rwxrwxrwx 1 root root 0 May 9 13:19 f2a -rwxrwxrwx 1 root root 0 May 9 13:19 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0133483 s, 38.4 kB/s PASS 2e (6s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 13:20:11 (1778347211) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (3s) == sanityn test 2g: check blocks update on sync write ==== 13:20:14 (1778347214) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.393356 s, 5.3 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (5s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 13:20:20 (1778347220) PASS 3 (6s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 13:20:25 (1778347225) PASS 4 (6s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 13:20:31 (1778347231) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (4s) == sanityn test 6: remove of open file on other node ============================================================================ 13:20:35 (1778347235) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (5s) == sanityn test 7: remove of open directory on other node ======================================================================= 13:20:40 (1778347240) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (4s) == sanityn test 8: remove of open special file on other node ==================================================================== 13:20:44 (1778347244) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (4s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 13:20:48 (1778347248) PASS 9a (6s) == sanityn test 9b: append to striped sparse file ======== 13:20:54 (1778347254) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382532 s, 27.4 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0340595 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (4s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 13:20:58 (1778347258) 1+0 records in 1+0 records out 1 byte copied, 0.00433213 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0149183 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0064784 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0070864 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00964238 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0116876 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0135146 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0146625 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0136383 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00847584 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.018276 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00967749 s, 0.1 kB/s PASS 10a (5s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 13:21:03 (1778347263) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0129615 s, 237 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0566725 s, 72.3 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000283652 s, 10.8 MB/s PASS 10b (5s) == sanityn test 11: execution of file opened for write should return error ============================================================== 13:21:08 (1778347268) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7598 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 300: /mnt/lustre2/d11/f: Text file busy PASS 11 (5s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 13:21:13 (1778347273) warning: '-runtime' deprecated, use '-t runtime' instead - link 609 (time 1778347284.99 total 10.01 last 60.86) - link 1537 (time 1778347294.99 total 20.01 last 92.77) - link 2415 (time 1778347305.00 total 30.02 last 87.73) - link 2993 (time 1778347315.01 total 40.03 last 57.72) - link 3773 (time 1778347325.01 total 50.03 last 77.99) - link 4871 (time 1778347335.02 total 60.04 last 109.71) - link 6057 (time 1778347345.03 total 70.05 last 118.47) - link 6764 (time 1778347355.04 total 80.06 last 70.65) - link 7503 (time 1778347365.04 total 90.06 last 73.90) total: 8403 link in 100.00 seconds: 84.03 ops/second using seed 649748082 running for 100 seconds - stat 9157 (time 1778347287 ; total 11 ; last 11) - stat 10000 (time 1778347287 ; total 11 ; last 0) - stat 20000 (time 1778347297 ; total 21 ; last 10) - stat 28950 (time 1778347308 ; total 32 ; last 11) - stat 30000 (time 1778347309 ; total 33 ; last 1) - stat 39514 (time 1778347320 ; total 44 ; last 11) - stat 40000 (time 1778347320 ; total 44 ; last 0) - stat 47916 (time 1778347331 ; total 55 ; last 11) - stat 50000 (time 1778347335 ; total 59 ; last 4) - stat 54654 (time 1778347346 ; total 70 ; last 11) - stat 58933 (time 1778347357 ; total 81 ; last 11) - stat 60000 (time 1778347359 ; total 83 ; last 2) - stat 63867 (time 1778347370 ; total 94 ; last 11) total: 66994 stats in 100 seconds: 669.940002 stats/second - unlinked 0 (time 1778347326 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile8403) error: No such file or directory total: 8402 unlinks in 73 seconds: 115.095894 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21384) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21386) - No such process PASS 12 (128s) == sanityn test 13: test directory page revocation ======= 13:23:21 (1778347401) f13 f13 PASS 13 (3s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 13:23:24 (1778347404) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7598 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 335: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 22567 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (3s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 13:23:27 (1778347407) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 23145 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (2s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 13:23:29 (1778347409) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 350: 23723 Terminated $DIR1/$tdir/sleep 60 PASS 14b (2s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 13:23:31 (1778347411) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 362: 24304 Terminated $DIR1/$tdir/sleep 60 PASS 14c (3s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 13:23:34 (1778347414) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 373: 24886 Terminated $DIR1/$tdir/sleep 60 PASS 14d (3s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 13:23:37 (1778347417) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7527424 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7527424KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (30s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 13:24:07 (1778347447) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0991048 s, 84.6 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.067385 s, 124 MB/s Chance of close/open is 1 in 50 Seed set to 7448 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read skipping zero size read 000011[0] 1778347448.763405 write 0x778acf thru 0x786805 (0xdd37 bytes) truncating to largest ever: 0x5e2d90 000013[1] 1778347448.769960 trunc from 0x786806 to 0x5e2d8f (0x1a3a76 bytes) 000014[1] 1778347448.780478 trunc from 0x5e2d90 to 0x255448 (0x38d947 bytes) 000015[1] 1778347448.783621 mapread 0x0e37c1 thru 0x0e6065 (0x28a5 bytes) 000020[1] 1778347448.805274 mapread 0x17e3ef thru 0x183bbd (0x57cf bytes) 000022[1] 1778347448.812556 read 0x1cef3b thru 0x1da562 (0xb628 bytes) 000023[0] 1778347448.818803 read 0x197d82 thru 0x19d414 (0x5693 bytes) 000024[1] 1778347448.826914 read 0x18ab6e thru 0x190ed1 (0x6364 bytes) 000025[1] 1778347448.845866 mapwrite 0x5b5d7b thru 0x5c20c2 (0xc348 bytes) 000027[1] 1778347448.895342 read 0x2b35b1 thru 0x2c29b9 (0xf409 bytes) 000029[1] 1778347448.903624 mapread 0x138899 thru 0x147d46 (0xf4ae bytes) 000030[0] 1778347448.917767 write 0x940900 thru 0x9488ec (0x7fed bytes) 000032[1] 1778347448.932137 mapwrite 0x84ad6a thru 0x851de9 (0x7080 bytes) 000036[0] 1778347448.961062 mapread 0x83bae1 thru 0x846c88 (0xb1a8 bytes) 000037[1] 1778347448.973879 mapread 0x6ed348 thru 0x6f6a4d (0x9706 bytes) 000039[0] 1778347448.986475 read 0x150f52 thru 0x153ab7 (0x2b66 bytes) 000040[0] 1778347448.991933 mapread 0x47d207 thru 0x48a4d8 (0xd2d2 bytes) 000041[1] 1778347449.005304 mapread 0x3a13fd thru 0x3ab26d (0x9e71 bytes) 000043[1] 1778347449.011417 mapwrite 0x472c05 thru 0x4769e9 (0x3de5 bytes) 000044[1] 1778347449.029144 mapwrite 0x86d656 thru 0x87a8c3 (0xd26e bytes) 000045[1] 1778347449.055296 mapread 0x62fb01 thru 0x63dc33 (0xe133 bytes) truncating to largest ever: 0x74d8b5 000048[1] 1778347449.081113 trunc from 0x9488ed to 0x74d8b4 (0x1fb038 bytes) 000049[0] 1778347449.087746 mapread 0x23d9c8 thru 0x240f0a (0x3543 bytes) 000051[1] 1778347449.104304 write 0x6b70be thru 0x6c6a5d (0xf9a0 bytes) 000053[1] 1778347449.113055 trunc from 0x74d8b5 to 0x6363f2 (0x1174c2 bytes) 000054[0] 1778347449.125625 mapwrite 0x68cfe7 thru 0x695d93 (0x8dad bytes) 000056[1] 1778347449.160446 mapwrite 0x2ddc39 thru 0x2eca42 (0xee0a bytes) 000057[1] 1778347449.188134 trunc from 0x695d94 to 0x5a7513 (0xee880 bytes) 000061[0] 1778347449.195289 trunc from 0x5a7514 to 0x009d08 (0x59d80b bytes) truncating to largest ever: 0x9c3b30 000063[0] 1778347449.209349 trunc from 0x009d09 to 0x9c3b2f (0x9b9e27 bytes) 000064[0] 1778347449.217411 write 0x89f39b thru 0x8a168a (0x22f0 bytes) 000066[0] 1778347449.226652 write 0x12263a thru 0x12baba (0x9481 bytes) 000067[1] 1778347449.234824 write 0x02494a thru 0x0340d1 (0xf788 bytes) 000069[0] 1778347449.251360 mapwrite 0x2df531 thru 0x2e8b7f (0x964f bytes) 000071[1] 1778347449.282850 trunc from 0x9c3b30 to 0x52ea19 (0x495116 bytes) 000072[1] 1778347449.293264 trunc from 0x52ea1a to 0x82e68d (0x2ffc74 bytes) 000073[1] 1778347449.302764 mapwrite 0x38f60c thru 0x39da06 (0xe3fb bytes) 000074[1] 1778347449.329559 trunc from 0x82e68e to 0x0315ab (0x7fd0e2 bytes) 000075[1] 1778347449.339792 write 0x9c9cdb thru 0x9ca615 (0x93b bytes) 000077[1] 1778347449.345746 trunc from 0x9ca616 to 0x20ff15 (0x7ba700 bytes) 000078[1] 1778347449.352274 mapread 0x141956 thru 0x14368a (0x1d35 bytes) 000079[0] 1778347449.363272 trunc from 0x20ff16 to 0x07f11d (0x190df8 bytes) 000080[0] 1778347449.376829 trunc from 0x07f11e to 0x005fff (0x7911e bytes) 000082[0] 1778347449.389648 read 0x004dd3 thru 0x005852 (0xa80 bytes) 000084[0] 1778347449.397471 mapread 0x004236 thru 0x005fff (0x1dca bytes) 000086[1] 1778347449.398792 write 0x0e04ad thru 0x0eb21d (0xad71 bytes) 000087[1] 1778347449.406637 mapwrite 0x49c2c5 thru 0x4a1797 (0x54d3 bytes) 000088[0] 1778347449.440245 mapwrite 0x417c57 thru 0x42585a (0xdc04 bytes) 000091[0] 1778347449.500789 mapread 0x19ada0 thru 0x1a41e6 (0x9447 bytes) 000095[0] 1778347449.537237 write 0x0f8912 thru 0x107ac9 (0xf1b8 bytes) 000096[0] 1778347449.548475 mapread 0x3b4d0a thru 0x3c1457 (0xc74e bytes) 000098[0] 1778347449.573955 mapwrite 0x2df67d thru 0x2ec8df (0xd263 bytes) 000099[0] 1778347449.604995 mapwrite 0x8d7d10 thru 0x8dcadb (0x4dcc bytes) 000100[0] 1778347449.649788 trunc from 0x8dcadc to 0x825aa0 (0xb703b bytes) 000101[0] 1778347449.656022 trunc from 0x825aa1 to 0x851e0c (0x2c36c bytes) 000103[1] 1778347449.669801 read 0x4404ae thru 0x44c28b (0xbdde bytes) 000104[0] 1778347449.683257 mapread 0x518652 thru 0x520e0a (0x87b9 bytes) 000105[1] 1778347449.696000 read 0x15b0d3 thru 0x15f87f (0x47ad bytes) 000107[1] 1778347449.701144 mapwrite 0x5148e8 thru 0x518c3e (0x4357 bytes) 000109[0] 1778347449.728322 read 0x32504b thru 0x333eb0 (0xee66 bytes) 000112[0] 1778347449.739956 write 0x9b0f40 thru 0x9b9b87 (0x8c48 bytes) 000115[1] 1778347449.756070 mapread 0x3f7a96 thru 0x3fe904 (0x6e6f bytes) 000119[0] 1778347449.779539 read 0x6c0f9d thru 0x6c3d63 (0x2dc7 bytes) 000121[0] 1778347449.785271 write 0x4a6aa9 thru 0x4b097d (0x9ed5 bytes) 000122[1] 1778347449.794112 trunc from 0x9b9b88 to 0x756e8a (0x262cfd bytes) 000126[1] 1778347449.807388 trunc from 0x756e8b to 0x0899dc (0x6cd4ae bytes) truncating to largest ever: 0x9fb7fc 000127[0] 1778347449.812011 trunc from 0x0899dd to 0x9fb7fb (0x971e1f bytes) 000129[1] 1778347449.831981 trunc from 0x9fb7fc to 0x65f61a (0x39c1e1 bytes) 000130[1] 1778347449.841603 mapwrite 0x10ad86 thru 0x10b955 (0xbd0 bytes) 000131[0] 1778347449.856924 trunc from 0x65f61b to 0x97ca62 (0x31d448 bytes) 000133[0] 1778347449.874364 read 0x58a650 thru 0x597dd5 (0xd786 bytes) 000135[0] 1778347449.884335 read 0x8350a1 thru 0x837b4f (0x2aaf bytes) 000136[0] 1778347449.889777 mapread 0x907aa8 thru 0x914b15 (0xd06e bytes) 000140[0] 1778347449.904139 mapwrite 0x8faa96 thru 0x909b6a (0xf0d5 bytes) 000146[1] 1778347449.926677 read 0x4abca1 thru 0x4b708c (0xb3ec bytes) 000147[0] 1778347449.931826 trunc from 0x97ca63 to 0x2749c1 (0x7080a1 bytes) 000148[0] 1778347449.938894 mapwrite 0x3f5733 thru 0x3fe55b (0x8e29 bytes) 000149[0] 1778347449.965023 trunc from 0x3fe55c to 0x23aa4a (0x1c3b11 bytes) 000150[1] 1778347449.970471 write 0x6c00de thru 0x6c0f7f (0xea2 bytes) 000151[0] 1778347449.978287 write 0x4cf1a6 thru 0x4d2102 (0x2f5d bytes) 000152[1] 1778347449.985600 read 0x35c00f thru 0x36b761 (0xf753 bytes) 000156[1] 1778347450.013277 write 0x6dc64c thru 0x6e62e8 (0x9c9d bytes) 000159[1] 1778347450.022777 mapread 0x249ca2 thru 0x25869e (0xe9fd bytes) 000163[1] 1778347450.047856 mapwrite 0x02a117 thru 0x039346 (0xf230 bytes) 000164[0] 1778347450.073035 write 0x002c44 thru 0x0042f9 (0x16b6 bytes) 000167[0] 1778347450.089740 write 0x5e061a thru 0x5e935c (0x8d43 bytes) 000170[1] 1778347450.102572 mapread 0x304579 thru 0x309e8b (0x5913 bytes) 000171[1] 1778347450.129290 read 0x06ec0a thru 0x06f01c (0x413 bytes) 000172[0] 1778347450.134141 mapread 0x3e74ad thru 0x3f10fd (0x9c51 bytes) 000175[0] 1778347450.156177 trunc from 0x6e62e9 to 0x4700fe (0x2761ea bytes) 000176[0] 1778347450.167497 mapread 0x2310c7 thru 0x237828 (0x6762 bytes) 000177[0] 1778347450.177678 mapwrite 0x68a8cc thru 0x6939d4 (0x9109 bytes) 000178[0] 1778347450.206347 write 0x8a0524 thru 0x8a1274 (0xd51 bytes) 000184[0] 1778347450.225539 read 0x0c685a thru 0x0cfdab (0x9552 bytes) 000185[0] 1778347450.228665 read 0x2f640f thru 0x3051f5 (0xede7 bytes) 000187[0] 1778347450.235541 mapwrite 0x47c5c3 thru 0x488374 (0xbdb2 bytes) 000188[0] 1778347450.255102 mapread 0x5c3586 thru 0x5ca70e (0x7189 bytes) 000189[1] 1778347450.264950 read 0x26f702 thru 0x279a09 (0xa308 bytes) 000191[0] 1778347450.279076 mapread 0x3092e5 thru 0x312967 (0x9683 bytes) 000193[1] 1778347450.289948 trunc from 0x8a1275 to 0x0ec6db (0x7b4b99 bytes) 000194[1] 1778347450.299692 mapwrite 0x93c9dd thru 0x94a879 (0xde9d bytes) 000195[1] 1778347450.325264 mapwrite 0x7983f4 thru 0x79e7e7 (0x63f4 bytes) 000196[0] 1778347450.342856 mapwrite 0x168a14 thru 0x171ecd (0x94ba bytes) 000201[1] 1778347450.381168 trunc from 0x94a87a to 0x8708bc (0xd9fbd bytes) 000203[1] 1778347450.394383 trunc from 0x8708bd to 0x056377 (0x81a545 bytes) 000206[0] 1778347450.399183 mapread 0x0140e6 thru 0x01eaa3 (0xa9be bytes) 000207[0] 1778347450.411269 mapread 0x02c5f5 thru 0x035281 (0x8c8d bytes) 000211[1] 1778347450.424756 write 0x272a32 thru 0x279d22 (0x72f1 bytes) 000212[1] 1778347450.431563 read 0x1d8dec thru 0x1df502 (0x6717 bytes) 000216[1] 1778347450.447432 trunc from 0x279d23 to 0x7d36dc (0x5599ba bytes) 000220[1] 1778347450.467301 read 0x36d2e4 thru 0x3729aa (0x56c7 bytes) 000222[1] 1778347450.471413 write 0x514db4 thru 0x51c27d (0x74ca bytes) 000223[0] 1778347450.474376 read 0x272f94 thru 0x27557b (0x25e8 bytes) 000224[0] 1778347450.484436 read 0x68b0ba thru 0x693349 (0x8290 bytes) 000226[1] 1778347450.491394 read 0x1ab472 thru 0x1ad492 (0x2021 bytes) 000228[1] 1778347450.493114 mapwrite 0x2e6c6e thru 0x2ec99a (0x5d2d bytes) 000230[1] 1778347450.515278 mapread 0x453ac2 thru 0x455511 (0x1a50 bytes) 000231[0] 1778347450.520080 mapread 0x4598f9 thru 0x45cfe2 (0x36ea bytes) 000232[1] 1778347450.526247 mapread 0x274558 thru 0x27b8ce (0x7377 bytes) 000234[0] 1778347450.537387 mapwrite 0x33db92 thru 0x33e0bc (0x52b bytes) 000236[1] 1778347450.557159 trunc from 0x7d36dd to 0x62748d (0x1ac24f bytes) 000237[0] 1778347450.569514 mapread 0x579b5e thru 0x583240 (0x96e3 bytes) 000239[0] 1778347450.580147 mapread 0x44a97b thru 0x450781 (0x5e07 bytes) 000242[1] 1778347450.593860 read 0x3f8f49 thru 0x404daf (0xbe67 bytes) 000243[0] 1778347450.600035 trunc from 0x62748e to 0x7cd88f (0x1a6402 bytes) 000244[1] 1778347450.613031 mapread 0x082f58 thru 0x085cef (0x2d98 bytes) 000245[1] 1778347450.624990 write 0x98b465 thru 0x9903e1 (0x4f7d bytes) 000246[0] 1778347450.632171 write 0x7deeb5 thru 0x7e014e (0x129a bytes) 000247[0] 1778347450.636279 write 0x2e1787 thru 0x2e60ea (0x4964 bytes) 000248[1] 1778347450.645883 write 0x1cbc31 thru 0x1d3cb0 (0x8080 bytes) 000252[1] 1778347450.664392 write 0x4178b4 thru 0x421991 (0xa0de bytes) 000254[1] 1778347450.674536 trunc from 0x9903e2 to 0x0013c5 (0x98f01c bytes) 000256[0] 1778347450.684571 mapread 0x000a87 thru 0x000ff3 (0x56d bytes) 000260[1] 1778347450.697432 read 0x0010ab thru 0x0013c5 (0x31b bytes) 000264[1] 1778347450.705668 mapread 0x000d39 thru 0x0013c5 (0x68d bytes) 000266[0] 1778347450.706686 mapwrite 0x934332 thru 0x93b323 (0x6ff2 bytes) 000267[1] 1778347450.733681 mapwrite 0x1496aa thru 0x158695 (0xefec bytes) 000269[0] 1778347450.775286 mapwrite 0x654c6b thru 0x655597 (0x92d bytes) 000273[0] 1778347450.802584 trunc from 0x93b324 to 0x993014 (0x57cf1 bytes) 000274[1] 1778347450.816289 read 0x6b822a thru 0x6bdf50 (0x5d27 bytes) 000276[0] 1778347450.826380 read 0x8cc268 thru 0x8cf15a (0x2ef3 bytes) 000278[0] 1778347450.833918 write 0x7efcd5 thru 0x7fe428 (0xe754 bytes) 000280[1] 1778347450.845778 read 0x5aec06 thru 0x5b645f (0x785a bytes) 000281[1] 1778347450.853439 read 0x4d8507 thru 0x4dd5df (0x50d9 bytes) 000283[0] 1778347450.861433 trunc from 0x993015 to 0x78291a (0x2106fa bytes) 000284[0] 1778347450.872152 write 0x78c2d7 thru 0x79af8a (0xecb4 bytes) 000285[0] 1778347450.876113 read 0x0deadc thru 0x0eda7d (0xefa2 bytes) 000288[1] 1778347450.886761 mapwrite 0x5f634a thru 0x5fd0d3 (0x6d8a bytes) 000290[1] 1778347450.914389 read 0x5e4453 thru 0x5ee1e7 (0x9d95 bytes) 000291[1] 1778347450.919914 mapwrite 0x743422 thru 0x74d9e8 (0xa5c7 bytes) 000292[1] 1778347450.941746 trunc from 0x79af8b to 0x75afbe (0x3ffcc bytes) 000293[1] 1778347450.948626 write 0x84d0b3 thru 0x85067a (0x35c8 bytes) 000295[0] 1778347450.954338 mapwrite 0x9d1283 thru 0x9d84a0 (0x721e bytes) 000296[1] 1778347450.984103 write 0x89b3ec thru 0x89c6b2 (0x12c7 bytes) 000297[1] 1778347450.992014 read 0x98c194 thru 0x994343 (0x81b0 bytes) 000298[0] 1778347450.997150 mapwrite 0x9a75f9 thru 0x9b7253 (0xfc5b bytes) 000300[0] 1778347451.022731 write 0x87555c thru 0x877a50 (0x24f5 bytes) 000305[1] 1778347451.036635 mapread 0x4d8bf3 thru 0x4e65c2 (0xd9d0 bytes) 000307[0] 1778347451.051166 read 0x6f9609 thru 0x700ede (0x78d6 bytes) 000309[1] 1778347451.057721 write 0x7665dc thru 0x773a40 (0xd465 bytes) 000310[1] 1778347451.066007 write 0x78d145 thru 0x79c8b8 (0xf774 bytes) 000313[0] 1778347451.079600 mapread 0x715710 thru 0x724ff3 (0xf8e4 bytes) 000317[0] 1778347451.105061 mapwrite 0x767713 thru 0x768a08 (0x12f6 bytes) 000318[1] 1778347451.119414 trunc from 0x9d84a1 to 0x77f01f (0x259481 bytes) 000320[1] 1778347451.134158 mapread 0x33a117 thru 0x3482a1 (0xe18b bytes) 000322[0] 1778347451.139951 mapread 0x2e846c thru 0x2eb016 (0x2bab bytes) 000327[1] 1778347451.162944 read 0x4cfdf8 thru 0x4da04c (0xa255 bytes) 000329[1] 1778347451.168850 trunc from 0x77f020 to 0x5f13e9 (0x18dc36 bytes) 000330[1] 1778347451.176384 mapread 0x404024 thru 0x40d82c (0x9809 bytes) 000331[0] 1778347451.184052 write 0x194b95 thru 0x19c190 (0x75fc bytes) 000332[0] 1778347451.191094 read 0x48e026 thru 0x4909d5 (0x29b0 bytes) 000335[0] 1778347451.200682 mapwrite 0x19a429 thru 0x1a48f4 (0xa4cc bytes) 000337[0] 1778347451.220446 trunc from 0x5f13ea to 0x748510 (0x157127 bytes) 000340[1] 1778347451.232214 trunc from 0x748511 to 0x99132b (0x248e1b bytes) 000341[0] 1778347451.244444 read 0x1df5ac thru 0x1e19de (0x2433 bytes) 000345[1] 1778347451.256258 read 0x58adba thru 0x58e8e6 (0x3b2d bytes) 000347[1] 1778347451.261599 trunc from 0x99132c to 0x085fda (0x90b351 bytes) 000348[1] 1778347451.271444 read 0x06c6b0 thru 0x07c1d9 (0xfb2a bytes) 000349[0] 1778347451.279145 write 0x9c732a thru 0x9cb82d (0x4504 bytes) 000351[0] 1778347451.290812 read 0x345693 thru 0x346d53 (0x16c1 bytes) 000353[1] 1778347451.298573 write 0x1b006e thru 0x1b21bf (0x2152 bytes) 000356[1] 1778347451.319524 write 0x53e86a thru 0x5465e2 (0x7d79 bytes) 000359[0] 1778347451.335295 read 0x23f087 thru 0x23f24c (0x1c6 bytes) 000360[0] 1778347451.343959 trunc from 0x9cb82e to 0x28b429 (0x740404 bytes) 000362[0] 1778347451.362634 write 0x8938a4 thru 0x89e2f7 (0xaa54 bytes) 000364[1] 1778347451.373406 mapread 0x5947d6 thru 0x59c066 (0x7891 bytes) 000366[1] 1778347451.403977 write 0x2b80de thru 0x2c245f (0xa382 bytes) 000370[0] 1778347451.418430 write 0x72a93b thru 0x72e03c (0x3702 bytes) 000371[0] 1778347451.423748 trunc from 0x89e2f8 to 0x5e64bc (0x2b7e3b bytes) 000375[1] 1778347451.441846 trunc from 0x5e64bd to 0x9cea26 (0x3e856a bytes) 000376[1] 1778347451.455283 mapread 0x0975fa thru 0x09cc4a (0x5651 bytes) 000379[1] 1778347451.471152 write 0x782b55 thru 0x7843bb (0x1867 bytes) 000390[1] 1778347451.498784 trunc from 0x9cea27 to 0x35ddfa (0x670c2c bytes) 000391[1] 1778347451.507597 mapread 0x061cab thru 0x068213 (0x6569 bytes) 000392[1] 1778347451.517655 write 0x56977e thru 0x573dd6 (0xa659 bytes) 000400[0] 1778347451.535387 mapwrite 0x22c8f8 thru 0x22fecc (0x35d5 bytes) 000402[1] 1778347451.558497 mapwrite 0x12e41e thru 0x135c89 (0x786c bytes) 000405[0] 1778347451.589618 mapwrite 0x832694 thru 0x833a59 (0x13c6 bytes) 000407[1] 1778347451.613385 read 0x492fb6 thru 0x4a2860 (0xf8ab bytes) 000408[0] 1778347451.621102 mapread 0x607ce4 thru 0x616de1 (0xf0fe bytes) 000409[0] 1778347451.627889 mapread 0x5cab2a thru 0x5d3e26 (0x92fd bytes) 000410[0] 1778347451.634993 trunc from 0x833a5a to 0x3ea1c6 (0x449893 bytes) 000411[1] 1778347451.645127 write 0x364a3b thru 0x36a728 (0x5cee bytes) 000413[0] 1778347451.655471 write 0x45c5fc thru 0x460ca1 (0x46a6 bytes) 000416[0] 1778347451.666486 mapwrite 0x576550 thru 0x576847 (0x2f8 bytes) 000417[1] 1778347451.689672 mapwrite 0x7d3bab thru 0x7dc76e (0x8bc4 bytes) 000420[1] 1778347451.721553 mapread 0x1a894c thru 0x1ab6f3 (0x2da8 bytes) 000421[0] 1778347451.729245 mapwrite 0x6b3201 thru 0x6bb425 (0x8225 bytes) 000423[1] 1778347451.759867 mapwrite 0x5e9cb7 thru 0x5f55a1 (0xb8eb bytes) 000425[1] 1778347451.782708 mapread 0x4cb4bc thru 0x4d8c25 (0xd76a bytes) 000428[0] 1778347451.793501 trunc from 0x7dc76f to 0x50c085 (0x2d06e9 bytes) 000430[0] 1778347451.809546 mapread 0x0ac496 thru 0x0b7b3f (0xb6aa bytes) 000431[1] 1778347451.817272 write 0x8130ce thru 0x8151ac (0x20df bytes) 000432[1] 1778347451.823261 write 0x856fdc thru 0x860fcb (0x9ff0 bytes) 000433[0] 1778347451.827147 trunc from 0x860fcc to 0x6a224a (0x1bed81 bytes) 000435[0] 1778347451.843632 mapwrite 0x366c05 thru 0x36b656 (0x4a52 bytes) 000436[0] 1778347451.859426 mapread 0x29a3b0 thru 0x29ef15 (0x4b66 bytes) 000437[0] 1778347451.865432 read 0x64970f thru 0x64f088 (0x597a bytes) 000438[0] 1778347451.871044 mapwrite 0x6bb9e8 thru 0x6c626d (0xa886 bytes) 000440[0] 1778347451.900877 write 0x95e3ea thru 0x96afd3 (0xcbea bytes) 000441[1] 1778347451.903188 write 0x742b18 thru 0x74dbb1 (0xb09a bytes) 000442[1] 1778347451.908909 mapwrite 0x2f47ff thru 0x303d80 (0xf582 bytes) 000445[0] 1778347451.942700 trunc from 0x96afd4 to 0x6a8a0a (0x2c25c9 bytes) 000446[0] 1778347451.953433 read 0x49b72e thru 0x4aa885 (0xf158 bytes) 000447[0] 1778347451.956187 mapwrite 0x2f2d5d thru 0x2fd7c8 (0xaa6c bytes) 000448[0] 1778347451.980460 write 0x6c4c36 thru 0x6c99d1 (0x4d9c bytes) 000450[1] 1778347451.985468 read 0x30f95c thru 0x31a8bc (0xaf61 bytes) 000455[1] 1778347452.004120 trunc from 0x6c99d2 to 0x04fa29 (0x679fa8 bytes) 000456[1] 1778347452.020172 read 0x007eb0 thru 0x00a658 (0x27a9 bytes) 000457[0] 1778347452.027474 trunc from 0x04fa2a to 0x9521f4 (0x9027cb bytes) 000460[1] 1778347452.047255 mapwrite 0x595c6f thru 0x59e6a6 (0x8a38 bytes) 000461[1] 1778347452.079481 trunc from 0x9521f5 to 0x208004 (0x74a1f0 bytes) 000462[0] 1778347452.086901 trunc from 0x208005 to 0x1fa038 (0xdfcc bytes) 000464[0] 1778347452.102168 mapread 0x1951f3 thru 0x19e34e (0x915c bytes) 000465[1] 1778347452.107904 write 0x4d26c0 thru 0x4dc1da (0x9b1b bytes) 000467[0] 1778347452.118822 trunc from 0x4dc1db to 0x92ccc2 (0x450ae8 bytes) 000471[1] 1778347452.142303 mapwrite 0x0847a0 thru 0x0946d6 (0xff37 bytes) 000472[0] 1778347452.179801 trunc from 0x92ccc3 to 0x019367 (0x91395b bytes) 000475[1] 1778347452.194786 write 0x63f40f thru 0x6430a4 (0x3c96 bytes) 000476[1] 1778347452.202267 write 0x1526eb thru 0x1583aa (0x5cc0 bytes) 000478[1] 1778347452.211793 mapwrite 0x074fab thru 0x077e73 (0x2ec9 bytes) 000479[1] 1778347452.226823 trunc from 0x6430a5 to 0x826c01 (0x1e3b5d bytes) 000481[1] 1778347452.240242 trunc from 0x826c02 to 0x5c7e96 (0x25ed6b bytes) 000482[0] 1778347452.248803 read 0x0ec797 thru 0x0f7165 (0xa9cf bytes) 000483[0] 1778347452.255648 mapread 0x1ea850 thru 0x1f9bcc (0xf37d bytes) 000486[0] 1778347452.265933 read 0x1f59c7 thru 0x1fbd10 (0x634a bytes) 000487[1] 1778347452.269771 write 0x6fe134 thru 0x70d9dd (0xf8aa bytes) 000489[0] 1778347452.274621 mapwrite 0x45f776 thru 0x460ddc (0x1667 bytes) 000492[1] 1778347452.299338 mapread 0x6cee4b thru 0x6d6eca (0x8080 bytes) 000493[0] 1778347452.316904 write 0x74e967 thru 0x75591d (0x6fb7 bytes) 000494[0] 1778347452.326256 trunc from 0x75591e to 0x29d0c6 (0x4b8857 bytes) 000498[0] 1778347452.349404 mapread 0x263379 thru 0x27239d (0xf025 bytes) 000499[1] 1778347452.357375 read 0x2992a6 thru 0x29d0c6 (0x3e21 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 7452 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x91f2b6 000002[0] 1778347452.589512 trunc from 00000000 to 0x91f2b5 (0x91f2b6 bytes) 000005[1] 1778347452.601883 mapwrite 0x2cf000 thru 0x2d612d (0x712e bytes) 000008[1] 1778347452.620545 trunc from 0x91f2b6 to 0x1882bf (0x796ff6 bytes) 000011[1] 1778347452.629643 mapread 0x137000 thru 0x14410d (0xd10e bytes) 000012[0] 1778347452.636059 mapread 0x0c9000 thru 0x0ccfba (0x3fbb bytes) 000016[0] 1778347452.645553 write 0x136000 thru 0x13cddf (0x6de0 bytes) 000020[0] 1778347452.653946 mapwrite 0x239000 thru 0x2448be (0xb8bf bytes) 000021[0] 1778347452.676448 trunc from 0x2448bf to 0x811165 (0x5cc8a7 bytes) 000022[0] 1778347452.680894 write 0x610000 thru 0x613c41 (0x3c42 bytes) 000023[1] 1778347452.682486 write_OD 0x982000 thru 0x982fff (0x1000 bytes) 000024[0] 1778347452.695601 mapread 0x526000 thru 0x52da2a (0x7a2b bytes) 000025[0] 1778347452.700251 write 0x939000 thru 0x93fc66 (0x6c67 bytes) 000026[1] 1778347452.704164 mapread 0x618000 thru 0x619935 (0x1936 bytes) 000027[0] 1778347452.711274 mapwrite 0x2d1000 thru 0x2d4d95 (0x3d96 bytes) 000029[1] 1778347452.734541 read_OD 0x797000 thru 0x7a1fff (0xb000 bytes) 000032[0] 1778347452.741302 mapwrite 0x742000 thru 0x74c6d9 (0xa6da bytes) 000033[1] 1778347452.761870 mapread 0x01a000 thru 0x01ca55 (0x2a56 bytes) 000035[0] 1778347452.769657 mapwrite 0x841000 thru 0x846d54 (0x5d55 bytes) 000037[1] 1778347452.785031 write_OD 0x3de000 thru 0x3e8fff (0xb000 bytes) 000038[1] 1778347452.796879 trunc from 0x983000 to 0x8f10e1 (0x91f1e bytes) 000041[0] 1778347452.803359 mapread 0x1d0000 thru 0x1dcb41 (0xcb42 bytes) 000042[1] 1778347452.814039 mapwrite 0x93c000 thru 0x93e8cc (0x28cd bytes) 000043[0] 1778347452.836395 read 0x644000 thru 0x6520d6 (0xe0d7 bytes) 000045[1] 1778347452.840686 mapread 0x150000 thru 0x156cdc (0x6cdd bytes) 000047[1] 1778347452.845628 write_OD 0x412000 thru 0x420fff (0xf000 bytes) 000050[0] 1778347452.860736 trunc from 0x93e8cd to 0x667480 (0x2d744c bytes) 000051[0] 1778347452.869617 read 0x395000 thru 0x39e112 (0x9113 bytes) 000052[0] 1778347452.872585 mapread 0x4f5000 thru 0x4ffe16 (0xae17 bytes) 000053[1] 1778347452.876921 read_OD 0x5a7000 thru 0x5a9fff (0x3000 bytes) 000055[0] 1778347452.882415 read 0x3f7000 thru 0x4032c2 (0xc2c3 bytes) 000056[0] 1778347452.884351 read 0x2db000 thru 0x2df740 (0x4741 bytes) 000057[0] 1778347452.886131 mapwrite 0x9b2000 thru 0x9bb961 (0x9962 bytes) truncating to largest ever: 0x97021f 000059[0] 1778347452.909302 trunc from 0x9bb962 to 0x97021e (0x4b743 bytes) 000060[1] 1778347452.913136 mapwrite 0x48c000 thru 0x498ff3 (0xcff4 bytes) 000061[1] 1778347452.936441 trunc from 0x97021f to 0x560e9b (0x40f383 bytes) 000064[1] 1778347452.948464 mapwrite 0x484000 thru 0x492b1e (0xeb1f bytes) 000068[0] 1778347452.972600 write 0x449000 thru 0x44cb1a (0x3b1b bytes) 000072[0] 1778347452.979947 write 0x4fe000 thru 0x50cb90 (0xeb91 bytes) 000074[0] 1778347452.982063 mapread 0x082000 thru 0x08e97c (0xc97d bytes) 000075[0] 1778347452.990454 mapwrite 0x0fd000 thru 0x10ae05 (0xde06 bytes) 000076[1] 1778347453.009459 write_OD 0x252000 thru 0x25dfff (0xc000 bytes) 000077[1] 1778347453.024319 write_OD 0x876000 thru 0x878fff (0x3000 bytes) 000078[0] 1778347453.037813 write 0x2c2000 thru 0x2c621c (0x421d bytes) 000079[1] 1778347453.041480 mapread 0x84e000 thru 0x85724c (0x924d bytes) 000080[0] 1778347453.052136 mapwrite 0x4e2000 thru 0x4ec83e (0xa83f bytes) 000082[1] 1778347453.073348 read_OD 0x4f9000 thru 0x4fcfff (0x4000 bytes) 000083[1] 1778347453.078861 mapread 0x050000 thru 0x05517b (0x517c bytes) 000084[1] 1778347453.083500 mapwrite 0x751000 thru 0x758ed5 (0x7ed6 bytes) 000086[1] 1778347453.103319 write_OD 0x112000 thru 0x113fff (0x2000 bytes) 000088[1] 1778347453.116986 trunc from 0x879000 to 0x54509e (0x333f61 bytes) 000092[1] 1778347453.131745 mapwrite 0x63f000 thru 0x64ec49 (0xfc4a bytes) 000094[1] 1778347453.160462 trunc from 0x64ec4a to 0x50b56f (0x1436da bytes) 000096[0] 1778347453.170816 mapread 0x270000 thru 0x27546b (0x546c bytes) 000097[1] 1778347453.180481 mapread 0x2cb000 thru 0x2ccfe9 (0x1fea bytes) 000098[1] 1778347453.183239 mapwrite 0x812000 thru 0x81aee6 (0x8ee7 bytes) 000099[0] 1778347453.205283 trunc from 0x81aee7 to 0x3ac172 (0x46ed74 bytes) 000104[1] 1778347453.219309 read_OD 0x040000 thru 0x04cfff (0xd000 bytes) 000105[0] 1778347453.225089 write 0x6d9000 thru 0x6db2a5 (0x22a6 bytes) 000106[1] 1778347453.231044 write_OD 0x2c6000 thru 0x2cefff (0x9000 bytes) 000107[0] 1778347453.243521 mapread 0x024000 thru 0x02c07b (0x807c bytes) 000111[1] 1778347453.249550 read_OD 0x436000 thru 0x43efff (0x9000 bytes) 000112[0] 1778347453.251650 mapwrite 0x832000 thru 0x8360c4 (0x40c5 bytes) 000113[0] 1778347453.273608 trunc from 0x8360c5 to 0x51e448 (0x317c7c bytes) 000116[0] 1778347453.283995 read 0x297000 thru 0x2984d4 (0x14d5 bytes) 000121[0] 1778347453.295119 mapread 0x42b000 thru 0x4345cf (0x95d0 bytes) 000123[1] 1778347453.306384 mapread 0x254000 thru 0x258acb (0x4acc bytes) 000124[1] 1778347453.315189 write_OD 0x1ee000 thru 0x1f0fff (0x3000 bytes) 000127[0] 1778347453.327861 mapread 0x4a5000 thru 0x4a7b35 (0x2b36 bytes) 000128[0] 1778347453.330480 mapread 0x48f000 thru 0x4952d4 (0x62d5 bytes) 000129[1] 1778347453.332705 read_OD 0x0ca000 thru 0x0d0fff (0x7000 bytes) 000130[1] 1778347453.334981 write_OD 0x3a0000 thru 0x3acfff (0xd000 bytes) 000132[1] 1778347453.348378 mapread 0x01b000 thru 0x028a4b (0xda4c bytes) 000135[0] 1778347453.354838 mapread 0x1a6000 thru 0x1aadf0 (0x4df1 bytes) 000137[0] 1778347453.360650 trunc from 0x51e449 to 0x632a55 (0x11460d bytes) 000138[0] 1778347453.368462 write 0x821000 thru 0x82c432 (0xb433 bytes) 000139[1] 1778347453.369628 trunc from 0x82c433 to 0x7cfd8c (0x5c6a6 bytes) 000141[0] 1778347453.378043 mapread 0x584000 thru 0x586ef0 (0x2ef1 bytes) 000142[1] 1778347453.386965 read_OD 0x105000 thru 0x10bfff (0x7000 bytes) 000143[0] 1778347453.389911 trunc from 0x7cfd8d to 0x0890d7 (0x746cb5 bytes) 000148[1] 1778347453.408021 mapread 0x086000 thru 0x088705 (0x2706 bytes) 000153[1] 1778347453.415027 trunc from 0x0890d8 to 0x1d26a5 (0x1495ce bytes) 000154[1] 1778347453.424612 write_OD 0x29e000 thru 0x2a9fff (0xc000 bytes) 000156[1] 1778347453.437714 trunc from 0x2aa000 to 0x605ab5 (0x35bab6 bytes) 000157[0] 1778347453.444729 trunc from 0x605ab6 to 0x725ee3 (0x12042e bytes) 000161[0] 1778347453.451284 read 0x0c7000 thru 0x0cc892 (0x5893 bytes) 000163[1] 1778347453.453733 read_OD 0x6db000 thru 0x6e0fff (0x6000 bytes) 000167[1] 1778347453.461501 mapread 0x1b7000 thru 0x1ba0f9 (0x30fa bytes) 000169[0] 1778347453.467560 mapwrite 0x6eb000 thru 0x6f52b2 (0xa2b3 bytes) 000174[0] 1778347453.491222 write 0x29d000 thru 0x2a31bf (0x61c0 bytes) 000175[0] 1778347453.496173 write 0x2a8000 thru 0x2b5cf8 (0xdcf9 bytes) 000177[1] 1778347453.501921 read_OD 0x359000 thru 0x363fff (0xb000 bytes) 000180[0] 1778347453.510548 read 0x540000 thru 0x548794 (0x8795 bytes) 000187[0] 1778347453.515285 trunc from 0x725ee4 to 0x2ed0fb (0x438de8 bytes) 000191[0] 1778347453.529900 mapwrite 0x434000 thru 0x43b516 (0x7517 bytes) 000192[0] 1778347453.555279 mapread 0x30b000 thru 0x30c7ae (0x17af bytes) 000194[0] 1778347453.561733 read 0x128000 thru 0x12f772 (0x7773 bytes) 000195[0] 1778347453.563864 read 0x1ae000 thru 0x1bd27a (0xf27b bytes) 000197[0] 1778347453.566264 mapwrite 0x5e0000 thru 0x5e6d13 (0x6d14 bytes) 000199[1] 1778347453.587038 mapwrite 0x700000 thru 0x70c5a8 (0xc5a9 bytes) 000202[0] 1778347453.618414 read 0x551000 thru 0x556d33 (0x5d34 bytes) 000203[1] 1778347453.623314 mapread 0x39e000 thru 0x3a2fc3 (0x4fc4 bytes) 000207[1] 1778347453.628145 write_OD 0x4e4000 thru 0x4e4fff (0x1000 bytes) 000208[1] 1778347453.638597 write_OD 0x550000 thru 0x554fff (0x5000 bytes) 000211[1] 1778347453.651002 trunc from 0x70c5a9 to 0x2ef1bb (0x41d3ed bytes) 000213[0] 1778347453.657752 read 0x11d000 thru 0x11e7e2 (0x17e3 bytes) 000215[1] 1778347453.663030 mapread 0x1d9000 thru 0x1e561b (0xc61c bytes) 000220[0] 1778347453.670186 write 0x6ed000 thru 0x6fa630 (0xd631 bytes) 000221[0] 1778347453.676856 mapread 0x33c000 thru 0x3429b1 (0x69b2 bytes) 000223[0] 1778347453.682669 mapread 0x519000 thru 0x5267e9 (0xd7ea bytes) 000227[0] 1778347453.697332 mapread 0x2d8000 thru 0x2e7206 (0xf207 bytes) 000229[0] 1778347453.703312 read 0x58c000 thru 0x59a80c (0xe80d bytes) 000233[0] 1778347453.712335 trunc from 0x6fa631 to 0x8f9dde (0x1ff7ae bytes) 000236[1] 1778347453.721898 mapread 0x1b2000 thru 0x1c1052 (0xf053 bytes) 000238[1] 1778347453.732899 read_OD 0x49c000 thru 0x4a1fff (0x6000 bytes) 000241[1] 1778347453.738972 mapwrite 0x6c8000 thru 0x6c9cff (0x1d00 bytes) 000242[1] 1778347453.753291 write 0x777000 thru 0x77ca47 (0x5a48 bytes) 000243[1] 1778347453.758640 write 0x76d000 thru 0x7703ac (0x33ad bytes) 000249[1] 1778347453.774271 write 0x597000 thru 0x59fe9c (0x8e9d bytes) 000251[0] 1778347453.776108 write 0x23d000 thru 0x241fe2 (0x4fe3 bytes) 000256[1] 1778347453.788091 write 0x610000 thru 0x613e96 (0x3e97 bytes) 000258[0] 1778347453.799445 trunc from 0x8f9ddf to 0x317f4f (0x5e1e8f bytes) 000259[1] 1778347453.809436 write 0x292000 thru 0x2a1c64 (0xfc65 bytes) 000260[0] 1778347453.813107 read 0x1a2000 thru 0x1a494b (0x294c bytes) 000264[1] 1778347453.834655 mapread 0x220000 thru 0x22c042 (0xc043 bytes) 000266[1] 1778347453.839936 trunc from 0x317f50 to 0x3ae750 (0x96801 bytes) 000269[1] 1778347453.849319 trunc from 0x3ae751 to 0x792650 (0x3e3f00 bytes) 000271[0] 1778347453.852996 mapread 0x409000 thru 0x41891a (0xf91b bytes) 000273[1] 1778347453.874476 mapread 0x36b000 thru 0x376f25 (0xbf26 bytes) 000278[1] 1778347453.882939 write 0x976000 thru 0x97639d (0x39e bytes) 000281[1] 1778347453.896425 write 0x386000 thru 0x38f71c (0x971d bytes) 000283[0] 1778347453.899847 read 0x792000 thru 0x79c235 (0xa236 bytes) 000286[0] 1778347453.907082 mapwrite 0x96c000 thru 0x977b7d (0xbb7e bytes) 000287[1] 1778347453.930005 write 0x7fe000 thru 0x8008fb (0x28fc bytes) 000288[1] 1778347453.934567 mapread 0x096000 thru 0x097b88 (0x1b89 bytes) 000289[1] 1778347453.936871 mapread 0x96c000 thru 0x975b82 (0x9b83 bytes) 000291[1] 1778347453.944906 mapwrite 0x1c0000 thru 0x1cc773 (0xc774 bytes) 000292[0] 1778347453.966160 write 0x246000 thru 0x24c5eb (0x65ec bytes) 000293[0] 1778347453.972773 mapread 0x4ff000 thru 0x4fff17 (0xf18 bytes) 000297[0] 1778347453.982760 mapwrite 0x369000 thru 0x3720be (0x90bf bytes) 000299[1] 1778347454.004649 trunc from 0x977b7e to 0x96f417 (0x8766 bytes) 000300[0] 1778347454.013879 trunc from 0x96f418 to 0x43bd38 (0x5336df bytes) 000301[0] 1778347454.021326 trunc from 0x43bd39 to 0x647c00 (0x20bec8 bytes) 000306[1] 1778347454.035431 read 0x121000 thru 0x125e4e (0x4e4f bytes) 000307[1] 1778347454.040162 mapread 0x50f000 thru 0x510cb0 (0x1cb1 bytes) 000310[0] 1778347454.043022 mapwrite 0x9bb000 thru 0x9c29bf (0x79c0 bytes) 000312[1] 1778347454.065633 mapread 0x49a000 thru 0x4a832f (0xe330 bytes) 000314[0] 1778347454.075992 mapwrite 0x67d000 thru 0x68259c (0x559d bytes) 000316[0] 1778347454.096670 mapread 0x225000 thru 0x22b81b (0x681c bytes) 000317[1] 1778347454.104069 trunc from 0x9c29c0 to 0x81d088 (0x1a5937 bytes) 000320[0] 1778347454.111062 write 0x48b000 thru 0x4992b4 (0xe2b5 bytes) 000321[1] 1778347454.116263 trunc from 0x81d089 to 0x891765 (0x746dd bytes) 000323[0] 1778347454.130079 trunc from 0x891766 to 0x0e2f8e (0x7ae7d7 bytes) 000325[0] 1778347454.139914 write 0x1e5000 thru 0x1e68e6 (0x18e7 bytes) 000328[0] 1778347454.143906 write 0x107000 thru 0x113ba1 (0xcba2 bytes) 000329[0] 1778347454.145679 read 0x16b000 thru 0x1781b4 (0xd1b5 bytes) 000330[0] 1778347454.153282 mapwrite 0x61c000 thru 0x61e873 (0x2874 bytes) 000331[1] 1778347454.188555 write 0x690000 thru 0x690007 (0x8 bytes) 000332[1] 1778347454.199826 read 0x5d7000 thru 0x5da7f4 (0x37f5 bytes) 000333[0] 1778347454.202702 trunc from 0x690008 to 0x9311a1 (0x2a119a bytes) 000335[1] 1778347454.214358 write 0x485000 thru 0x494d8e (0xfd8f bytes) 000338[0] 1778347454.218652 mapread 0x474000 thru 0x47771b (0x371c bytes) 000340[0] 1778347454.229772 read 0x0ce000 thru 0x0dc58e (0xe58f bytes) 000341[0] 1778347454.232296 write 0x2a3000 thru 0x2ad39d (0xa39e bytes) 000342[0] 1778347454.236392 write 0x906000 thru 0x90df43 (0x7f44 bytes) 000343[0] 1778347454.238057 trunc from 0x9311a2 to 0x75bee6 (0x1d52bb bytes) 000345[1] 1778347454.242873 write 0x473000 thru 0x477576 (0x4577 bytes) 000349[1] 1778347454.258677 read 0x066000 thru 0x06ec63 (0x8c64 bytes) 000350[1] 1778347454.263922 trunc from 0x75bee7 to 0x309976 (0x452570 bytes) 000351[1] 1778347454.271290 mapwrite 0x7df000 thru 0x7e6d39 (0x7d3a bytes) 000352[0] 1778347454.291790 trunc from 0x7e6d3a to 0x128a8b (0x6be2ae bytes) 000353[0] 1778347454.297953 mapread 0x04f000 thru 0x0563cd (0x73ce bytes) 000356[1] 1778347454.309774 read 0x02c000 thru 0x02c52a (0x52b bytes) 000358[0] 1778347454.320424 trunc from 0x128a8c to 0x029098 (0xff9f3 bytes) 000359[0] 1778347454.330908 write 0x810000 thru 0x81be61 (0xbe62 bytes) 000360[1] 1778347454.337097 write 0x61e000 thru 0x6266fa (0x86fb bytes) 000363[1] 1778347454.363357 mapwrite 0x599000 thru 0x59c29d (0x329e bytes) 000364[0] 1778347454.385585 mapwrite 0x9d8000 thru 0x9d8d2c (0xd2d bytes) 000366[1] 1778347454.415889 trunc from 0x9d8d2d to 0x0d0a65 (0x9082c7 bytes) 000367[1] 1778347454.421840 mapread 0x08a000 thru 0x08f613 (0x5614 bytes) 000368[0] 1778347454.428597 mapread 0x004000 thru 0x010363 (0xc364 bytes) 000371[0] 1778347454.439750 read 0x045000 thru 0x052fc1 (0xdfc2 bytes) 000375[0] 1778347454.445715 write 0x374000 thru 0x37f2b2 (0xb2b3 bytes) 000377[1] 1778347454.450307 mapread 0x2aa000 thru 0x2ac5df (0x25e0 bytes) 000379[1] 1778347454.461043 mapwrite 0x96b000 thru 0x97a8c2 (0xf8c3 bytes) 000381[0] 1778347454.491950 trunc from 0x97a8c3 to 0x248306 (0x7325bc bytes) 000383[0] 1778347454.505477 read 0x0dd000 thru 0x0e0f1a (0x3f1b bytes) 000384[0] 1778347454.516086 mapwrite 0x444000 thru 0x447e28 (0x3e29 bytes) 000386[0] 1778347454.548081 mapread 0x2e1000 thru 0x2e8362 (0x7363 bytes) 000388[0] 1778347454.554369 read 0x230000 thru 0x232430 (0x2431 bytes) 000393[1] 1778347454.567843 write 0x765000 thru 0x76eb4e (0x9b4f bytes) truncating to largest ever: 0x981ab6 000394[0] 1778347454.578133 trunc from 0x76eb4f to 0x981ab5 (0x212f67 bytes) 000396[1] 1778347454.592230 write 0x451000 thru 0x456194 (0x5195 bytes) 000399[1] 1778347454.596811 trunc from 0x981ab6 to 0x19fd91 (0x7e1d24 bytes) 000400[0] 1778347454.599902 write 0x50c000 thru 0x510e56 (0x4e57 bytes) 000401[1] 1778347454.606064 write 0x353000 thru 0x35a748 (0x7749 bytes) 000403[0] 1778347454.618285 mapwrite 0x628000 thru 0x628858 (0x859 bytes) 000404[1] 1778347454.639422 mapread 0x355000 thru 0x357f94 (0x2f95 bytes) 000406[1] 1778347454.648526 mapwrite 0x6b2000 thru 0x6b86ba (0x66bb bytes) 000407[1] 1778347454.675814 mapwrite 0x789000 thru 0x78d024 (0x4025 bytes) 000408[0] 1778347454.696433 trunc from 0x78d025 to 0x649a99 (0x14358b bytes) 000410[1] 1778347454.706732 mapread 0x2bf000 thru 0x2cdd78 (0xed79 bytes) 000412[1] 1778347454.727823 write 0x4fd000 thru 0x50b176 (0xe177 bytes) 000414[1] 1778347454.735450 mapwrite 0x3a4000 thru 0x3b24ad (0xe4ae bytes) 000416[0] 1778347454.769393 mapwrite 0x081000 thru 0x090f77 (0xff78 bytes) 000418[1] 1778347454.794842 mapread 0x30b000 thru 0x319b58 (0xeb59 bytes) 000419[1] 1778347454.821968 write 0x132000 thru 0x1383fc (0x63fd bytes) 000421[1] 1778347454.826513 mapread 0x341000 thru 0x34941d (0x841e bytes) 000424[1] 1778347454.839893 write 0x70e000 thru 0x71ac01 (0xcc02 bytes) 000425[1] 1778347454.840880 write 0x610000 thru 0x61b2f4 (0xb2f5 bytes) 000426[0] 1778347454.842561 read 0x5ff000 thru 0x609079 (0xa07a bytes) 000427[0] 1778347454.850270 read 0x54a000 thru 0x557fc0 (0xdfc1 bytes) 000428[1] 1778347454.854128 trunc from 0x71ac02 to 0x2a5782 (0x47547f bytes) 000431[0] 1778347454.865569 trunc from 0x2a5783 to 0x414171 (0x16e9ef bytes) 000432[0] 1778347454.872422 read 0x1c4000 thru 0x1c79d6 (0x39d7 bytes) 000433[0] 1778347454.874072 mapwrite 0x1b7000 thru 0x1c4a27 (0xda28 bytes) 000435[1] 1778347454.900638 mapwrite 0x205000 thru 0x209226 (0x4227 bytes) 000437[1] 1778347454.922338 mapwrite 0x223000 thru 0x22681c (0x381d bytes) 000439[1] 1778347454.948491 mapread 0x20a000 thru 0x20d32f (0x3330 bytes) 000440[1] 1778347454.953495 mapread 0x15f000 thru 0x164bb5 (0x5bb6 bytes) 000441[1] 1778347454.956019 trunc from 0x414172 to 0x42c8ca (0x18759 bytes) 000442[0] 1778347454.963511 write 0x35d000 thru 0x363d1e (0x6d1f bytes) 000446[1] 1778347454.967806 trunc from 0x42c8cb to 0x53ba2f (0x10f165 bytes) 000448[0] 1778347454.980382 mapread 0x465000 thru 0x4653d8 (0x3d9 bytes) 000449[1] 1778347454.986133 mapwrite 0x40e000 thru 0x415741 (0x7742 bytes) 000453[0] 1778347455.014761 read 0x472000 thru 0x480f06 (0xef07 bytes) 000454[0] 1778347455.020415 trunc from 0x53ba30 to 0x14bb70 (0x3efebf bytes) 000456[1] 1778347455.030076 mapwrite 0x2bc000 thru 0x2c86c4 (0xc6c5 bytes) 000458[1] 1778347455.059627 trunc from 0x2c86c5 to 0x2b16c7 (0x16ffd bytes) 000460[0] 1778347455.068081 write 0x33d000 thru 0x3443e3 (0x73e4 bytes) 000461[1] 1778347455.071031 mapwrite 0x405000 thru 0x4054ca (0x4cb bytes) 000463[0] 1778347455.093163 write 0x80d000 thru 0x814aea (0x7aeb bytes) 000464[1] 1778347455.097150 write 0x84f000 thru 0x84f850 (0x851 bytes) 000466[1] 1778347455.101759 mapwrite 0x6df000 thru 0x6e6782 (0x7783 bytes) 000467[1] 1778347455.117918 mapread 0x7c4000 thru 0x7c7969 (0x396a bytes) 000469[1] 1778347455.127220 mapread 0x144000 thru 0x14c9b6 (0x89b7 bytes) 000470[0] 1778347455.135281 trunc from 0x84f851 to 0x2f11f3 (0x55e65d bytes) 000472[0] 1778347455.146709 mapread 0x23e000 thru 0x24c350 (0xe351 bytes) 000474[1] 1778347455.156879 mapwrite 0x95a000 thru 0x95a111 (0x112 bytes) 000475[0] 1778347455.178403 mapread 0x5a2000 thru 0x5ae776 (0xc777 bytes) 000477[1] 1778347455.191043 trunc from 0x95a112 to 0x84850c (0x111c05 bytes) 000478[0] 1778347455.201922 mapread 0x6ab000 thru 0x6acd69 (0x1d6a bytes) 000479[1] 1778347455.210769 trunc from 0x84850d to 0x46d04f (0x3db4bd bytes) 000480[0] 1778347455.221382 mapread 0x2d2000 thru 0x2d987f (0x7880 bytes) 000481[1] 1778347455.232062 trunc from 0x46d050 to 0x69988f (0x22c840 bytes) 000482[1] 1778347455.242493 mapwrite 0x62c000 thru 0x6320a1 (0x60a2 bytes) 000484[0] 1778347455.265707 mapread 0x4a1000 thru 0x4a40d5 (0x30d6 bytes) 000485[1] 1778347455.275497 mapwrite 0x766000 thru 0x76cbe5 (0x6be6 bytes) 000487[0] 1778347455.300774 mapread 0x10b000 thru 0x11458c (0x958d bytes) 000488[1] 1778347455.309631 write 0x734000 thru 0x736400 (0x2401 bytes) 000489[0] 1778347455.318451 mapread 0x72d000 thru 0x72e028 (0x1029 bytes) 000491[0] 1778347455.327720 mapwrite 0x975000 thru 0x982e7a (0xde7b bytes) 000496[1] 1778347455.369174 read 0x7db000 thru 0x7de573 (0x3574 bytes) 000497[1] 1778347455.374353 mapwrite 0x513000 thru 0x51d37c (0xa37d bytes) 000498[0] 1778347455.397268 mapwrite 0x4a5000 thru 0x4ade22 (0x8e23 bytes) All operations completed A-OK! PASS 16a (9s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 13:24:16 (1778347456) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.107122 s, 78.3 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0733856 s, 114 MB/s Chance of close/open is 1 in 50 Seed set to 7457 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read skipping zero size read skipping zero size read skipping zero size read skipping zero size read truncating to largest ever: 0x907b06 000022[1] 1778347457.869691 trunc from 00000000 to 0x907b05 (0x907b06 bytes) 000029[0] 1778347457.901867 mapwrite 0x02c321 thru 0x02d338 (0x1018 bytes) 000032[1] 1778347457.931267 read 0x7df780 thru 0x7e1394 (0x1c15 bytes) 000033[0] 1778347457.936289 trunc from 0x907b06 to 0x7be9aa (0x14915b bytes) 000034[0] 1778347457.943392 mapread 0x2e67ff thru 0x2e7167 (0x969 bytes) 000035[0] 1778347457.946441 mapwrite 0x169516 thru 0x16b31b (0x1e06 bytes) 000037[0] 1778347457.966560 write 0x07267a thru 0x073aab (0x1432 bytes) 000043[1] 1778347457.988492 mapwrite 0x80d03e thru 0x80e781 (0x1744 bytes) 000046[1] 1778347458.022146 read 0x0ab1aa thru 0x0ab4eb (0x342 bytes) 000047[0] 1778347458.028851 trunc from 0x80e782 to 0x7a1c45 (0x6cb3c bytes) 000048[1] 1778347458.043305 write 0x5de673 thru 0x5def56 (0x8e4 bytes) 000051[0] 1778347458.060397 read 0x68a21a thru 0x68b60a (0x13f1 bytes) 000054[0] 1778347458.066772 mapwrite 0x8cbc89 thru 0x8cce74 (0x11ec bytes) 000055[0] 1778347458.097050 mapread 0x5731f0 thru 0x573550 (0x361 bytes) 000056[0] 1778347458.103836 write 0x2510b5 thru 0x25133e (0x28a bytes) 000057[1] 1778347458.106984 write 0x463b77 thru 0x4644a4 (0x92e bytes) 000058[1] 1778347458.116055 mapwrite 0x66c9fd thru 0x66e0f8 (0x16fc bytes) 000059[0] 1778347458.139640 mapread 0x6882f4 thru 0x688717 (0x424 bytes) 000060[1] 1778347458.145024 trunc from 0x8cce75 to 0x19af8d (0x731ee7 bytes) truncating to largest ever: 0x98815f 000061[0] 1778347458.155511 trunc from 0x19af8e to 0x98815e (0x7ed1d1 bytes) 000062[1] 1778347458.166694 trunc from 0x98815f to 0x46078a (0x5279d4 bytes) 000063[1] 1778347458.176226 write 0x46c14a thru 0x46df47 (0x1dfe bytes) 000068[0] 1778347458.194683 mapread 0x2e9b5f thru 0x2e9ca1 (0x143 bytes) 000069[1] 1778347458.201224 read 0x1e6c3d thru 0x1e8894 (0x1c58 bytes) 000070[0] 1778347458.202957 read 0x048cdc thru 0x049e67 (0x118c bytes) 000071[0] 1778347458.223062 read 0x3f6cfc thru 0x3f776e (0xa73 bytes) 000074[1] 1778347458.232346 trunc from 0x46df48 to 0x7078cd (0x299986 bytes) 000076[0] 1778347458.244502 read 0x3f49b5 thru 0x3f626f (0x18bb bytes) 000078[1] 1778347458.249353 trunc from 0x7078ce to 0x202117 (0x5057b6 bytes) 000081[1] 1778347458.267459 trunc from 0x202118 to 0x531e0b (0x32fcf4 bytes) 000082[0] 1778347458.273134 mapwrite 0x889fe1 thru 0x88ad37 (0xd57 bytes) 000083[1] 1778347458.297678 write 0x7814b5 thru 0x782d40 (0x188c bytes) 000087[0] 1778347458.320263 read 0x8407f6 thru 0x841e59 (0x1664 bytes) 000089[1] 1778347458.330143 mapwrite 0x5819f3 thru 0x583547 (0x1b55 bytes) 000092[0] 1778347458.364279 mapread 0x4c508e thru 0x4c6e91 (0x1e04 bytes) 000093[0] 1778347458.376307 mapwrite 0x9d3fb1 thru 0x9d5ba3 (0x1bf3 bytes) 000097[1] 1778347458.410602 mapwrite 0x3c9959 thru 0x3cae79 (0x1521 bytes) 000098[1] 1778347458.433516 write 0x2a9be6 thru 0x2ab954 (0x1d6f bytes) 000099[1] 1778347458.439863 write 0x21b718 thru 0x21cc51 (0x153a bytes) 000100[1] 1778347458.446084 mapwrite 0x1ea58e thru 0x1eafea (0xa5d bytes) 000101[0] 1778347458.467415 read 0x58700a thru 0x5870b6 (0xad bytes) 000103[0] 1778347458.476119 mapread 0x007b6a thru 0x009a77 (0x1f0e bytes) 000104[1] 1778347458.489372 write 0x847ad7 thru 0x849553 (0x1a7d bytes) 000105[1] 1778347458.498327 mapwrite 0x681208 thru 0x682b4c (0x1945 bytes) 000106[0] 1778347458.522863 trunc from 0x9d5ba4 to 0x0d91db (0x8fc9c8 bytes) 000112[1] 1778347458.543603 mapwrite 0x9d345c thru 0x9d4561 (0x1106 bytes) 000113[1] 1778347458.571214 read 0x3ebe1d thru 0x3ed30b (0x14ef bytes) 000114[0] 1778347458.573304 mapwrite 0x184848 thru 0x186270 (0x1a29 bytes) 000115[0] 1778347458.597578 mapread 0x19c310 thru 0x19d0c0 (0xdb1 bytes) 000116[1] 1778347458.606565 trunc from 0x9d4562 to 0x6e253d (0x2f2024 bytes) 000117[0] 1778347458.613646 mapwrite 0x75b5ba thru 0x75bf7f (0x9c6 bytes) 000118[0] 1778347458.638923 trunc from 0x75bf80 to 0x2568ac (0x5056d3 bytes) 000121[0] 1778347458.653512 mapwrite 0x91469e thru 0x915e3b (0x179e bytes) 000126[1] 1778347458.679935 mapwrite 0x29b4d4 thru 0x29cb16 (0x1643 bytes) truncating to largest ever: 0x9b93a8 000127[1] 1778347458.700036 trunc from 0x915e3c to 0x9b93a7 (0xa356c bytes) 000128[0] 1778347458.707330 read 0x6dc850 thru 0x6dce01 (0x5b2 bytes) 000129[0] 1778347458.715086 trunc from 0x9b93a8 to 0x8a7265 (0x112142 bytes) 000132[1] 1778347458.735566 write 0x48cde4 thru 0x48d91a (0xb37 bytes) 000137[1] 1778347458.759020 mapread 0x8452f2 thru 0x8472d5 (0x1fe4 bytes) 000139[0] 1778347458.768493 write 0x1c0abf thru 0x1c2435 (0x1977 bytes) 000141[1] 1778347458.781148 trunc from 0x8a7266 to 0x58108d (0x3261d8 bytes) 000142[1] 1778347458.793914 read 0x1cae53 thru 0x1cb1ee (0x39c bytes) 000144[0] 1778347458.797220 trunc from 0x58108e to 0x6aebab (0x12db1e bytes) 000145[0] 1778347458.807501 write 0x0ab4d8 thru 0x0ac12d (0xc56 bytes) 000146[0] 1778347458.810389 write 0x37e2a6 thru 0x37fea1 (0x1bfc bytes) 000147[0] 1778347458.818483 trunc from 0x6aebac to 0x894e71 (0x1e62c6 bytes) 000150[0] 1778347458.833826 mapwrite 0x90f86d thru 0x910539 (0xccd bytes) 000151[1] 1778347458.856353 read 0x345420 thru 0x345a3b (0x61c bytes) 000155[0] 1778347458.868713 read 0x563cce thru 0x5652be (0x15f1 bytes) 000157[1] 1778347458.877256 read 0x0fed05 thru 0x0ff54b (0x847 bytes) 000159[0] 1778347458.882485 mapread 0x098c99 thru 0x09a189 (0x14f1 bytes) 000161[1] 1778347458.891469 mapwrite 0x575361 thru 0x5760a0 (0xd40 bytes) 000162[1] 1778347458.909156 mapread 0x7b16ae thru 0x7b2940 (0x1293 bytes) 000163[1] 1778347458.916073 mapwrite 0x5fd395 thru 0x5fd450 (0xbc bytes) 000164[1] 1778347458.931110 read 0x4dbe95 thru 0x4dd0a0 (0x120c bytes) 000165[1] 1778347458.936524 read 0x3569ca thru 0x358365 (0x199c bytes) 000167[0] 1778347458.945540 trunc from 0x91053a to 0x8c9f07 (0x46632 bytes) 000168[0] 1778347458.959821 trunc from 0x8c9f08 to 0x5e1e9b (0x2e806c bytes) 000169[1] 1778347458.968698 mapwrite 0x577bbe thru 0x579098 (0x14db bytes) 000170[0] 1778347458.995046 trunc from 0x5e1e9c to 0x1678ca (0x47a5d1 bytes) 000171[0] 1778347459.003123 mapwrite 0x7962cc thru 0x7962f6 (0x2b bytes) 000175[0] 1778347459.033749 write 0x1a8616 thru 0x1aa395 (0x1d80 bytes) 000177[0] 1778347459.046860 mapwrite 0x2d3470 thru 0x2d3c94 (0x825 bytes) 000180[0] 1778347459.065862 read 0x61ae41 thru 0x61aef5 (0xb5 bytes) 000181[0] 1778347459.067941 read 0x56f8c3 thru 0x570d62 (0x14a0 bytes) 000182[1] 1778347459.074297 mapwrite 0x43ff6d thru 0x441152 (0x11e6 bytes) 000185[0] 1778347459.107517 trunc from 0x7962f7 to 0x80a9eb (0x746f5 bytes) 000188[0] 1778347459.121072 read 0x4299f4 thru 0x42b288 (0x1895 bytes) 000189[1] 1778347459.123315 mapread 0x0fa33a thru 0x0fb38e (0x1055 bytes) 000191[0] 1778347459.137139 trunc from 0x80a9ec to 0x694ff7 (0x1759f4 bytes) 000196[1] 1778347459.154975 write 0x238f38 thru 0x23a420 (0x14e9 bytes) 000200[1] 1778347459.175318 trunc from 0x694ff8 to 0x7b2543 (0x11d54c bytes) 000202[0] 1778347459.188984 trunc from 0x7b2544 to 0x5199e5 (0x298b5e bytes) 000204[1] 1778347459.209129 read 0x10e343 thru 0x1102d1 (0x1f8f bytes) 000206[0] 1778347459.221149 read 0x07c1a7 thru 0x07c7ae (0x608 bytes) 000207[1] 1778347459.224698 mapwrite 0x98f305 thru 0x990335 (0x1031 bytes) 000209[1] 1778347459.260913 write 0x93f111 thru 0x940dd3 (0x1cc3 bytes) 000210[0] 1778347459.265177 trunc from 0x990336 to 0x67e8a0 (0x311a95 bytes) 000212[1] 1778347459.285699 trunc from 0x67e8a1 to 0x303608 (0x37b298 bytes) 000213[1] 1778347459.294268 mapwrite 0x2674a9 thru 0x26759d (0xf5 bytes) 000214[0] 1778347459.308554 read 0x2e88ed thru 0x2e8c41 (0x355 bytes) 000215[0] 1778347459.318045 write 0x9e01e9 thru 0x9e1442 (0x125a bytes) 000216[1] 1778347459.326279 trunc from 0x9e1443 to 0x70a3b4 (0x2d708e bytes) 000218[0] 1778347459.344389 read 0x297b4f thru 0x2995ff (0x1ab1 bytes) 000220[1] 1778347459.353491 mapread 0x2a810e thru 0x2a86bc (0x5af bytes) 000221[1] 1778347459.355388 trunc from 0x70a3b5 to 0x63ec95 (0xcb71f bytes) 000222[0] 1778347459.367950 write 0x3c6dc9 thru 0x3c7e5f (0x1097 bytes) 000223[0] 1778347459.377088 read 0x3ab510 thru 0x3ab777 (0x268 bytes) 000229[0] 1778347459.401371 write 0x03c0e7 thru 0x03d81f (0x1739 bytes) 000230[1] 1778347459.407760 mapwrite 0x0761a0 thru 0x07735f (0x11c0 bytes) 000231[1] 1778347459.432212 read 0x3fca39 thru 0x3fde19 (0x13e1 bytes) 000232[0] 1778347459.437970 mapread 0x537cf3 thru 0x5388fa (0xc08 bytes) truncating to largest ever: 0x9d60d5 000236[0] 1778347459.448030 trunc from 0x63ec96 to 0x9d60d4 (0x39743f bytes) 000237[0] 1778347459.463902 mapwrite 0x00bf00 thru 0x00de70 (0x1f71 bytes) 000239[0] 1778347459.483681 mapwrite 0x61a6a5 thru 0x61b6d0 (0x102c bytes) 000243[1] 1778347459.514040 read 0x1a45f6 thru 0x1a4da1 (0x7ac bytes) 000246[1] 1778347459.523585 trunc from 0x9d60d5 to 0x715368 (0x2c0d6c bytes) 000247[1] 1778347459.537986 write 0x6dfeed thru 0x6e0122 (0x236 bytes) 000250[0] 1778347459.551051 read 0x31f95c thru 0x3205e0 (0xc85 bytes) 000252[1] 1778347459.556887 write 0x3eb3de thru 0x3ec3e1 (0x1004 bytes) 000255[1] 1778347459.567634 mapwrite 0x3db848 thru 0x3dc138 (0x8f1 bytes) 000256[1] 1778347459.584601 write 0x38b9a4 thru 0x38bb89 (0x1e6 bytes) 000258[1] 1778347459.588502 mapread 0x44b3a3 thru 0x44b5fd (0x25b bytes) 000259[1] 1778347459.596367 mapwrite 0x7ccbc2 thru 0x7ccddc (0x21b bytes) 000261[0] 1778347459.624174 trunc from 0x7ccddd to 0x390776 (0x43c666 bytes) 000263[1] 1778347459.641877 trunc from 0x390777 to 0x3161fc (0x7a57a bytes) 000264[1] 1778347459.651850 write 0x11a6e4 thru 0x11bbcb (0x14e8 bytes) 000265[1] 1778347459.660064 mapwrite 0x5f1a6c thru 0x5f34cb (0x1a60 bytes) 000269[0] 1778347459.694278 mapwrite 0x8df079 thru 0x8e03fa (0x1382 bytes) 000271[1] 1778347459.727562 read 0x20a917 thru 0x20c000 (0x16ea bytes) 000274[0] 1778347459.740554 read 0x682d84 thru 0x68301f (0x29c bytes) 000275[1] 1778347459.742471 mapread 0x070f82 thru 0x070fd8 (0x57 bytes) 000276[0] 1778347459.746074 mapread 0x86973b thru 0x86ae6d (0x1733 bytes) 000279[0] 1778347459.753647 trunc from 0x8e03fb to 0x424060 (0x4bc39a bytes) 000280[0] 1778347459.764970 read 0x29fee6 thru 0x2a05bf (0x6da bytes) 000281[0] 1778347459.772093 mapread 0x1c2458 thru 0x1c307a (0xc23 bytes) 000284[0] 1778347459.779827 trunc from 0x424061 to 0x56c1fd (0x14819d bytes) 000287[0] 1778347459.795163 write 0x431fc2 thru 0x43305b (0x109a bytes) 000288[1] 1778347459.798222 mapread 0x2924fe thru 0x293fb0 (0x1ab3 bytes) 000289[1] 1778347459.806739 read 0x147dd6 thru 0x148170 (0x39b bytes) 000291[0] 1778347459.836800 read 0x187a69 thru 0x188b78 (0x1110 bytes) 000292[0] 1778347459.838908 read 0x192376 thru 0x193347 (0xfd2 bytes) 000293[0] 1778347459.840728 trunc from 0x56c1fe to 0x9063c0 (0x39a1c3 bytes) 000297[0] 1778347459.860952 trunc from 0x9063c1 to 0x6fed30 (0x207690 bytes) 000301[0] 1778347459.873496 mapread 0x3cae3a thru 0x3cbf21 (0x10e8 bytes) 000304[0] 1778347459.881376 mapwrite 0x8e3ac9 thru 0x8e4b22 (0x105a bytes) 000305[1] 1778347459.900450 read 0x3eb686 thru 0x3eb938 (0x2b3 bytes) 000308[1] 1778347459.914747 trunc from 0x8e4b23 to 0x41396c (0x4d11b6 bytes) 000310[0] 1778347459.926622 write 0x4938b7 thru 0x493d2e (0x478 bytes) 000311[0] 1778347459.933222 mapwrite 0x945b83 thru 0x946b1c (0xf9a bytes) 000314[1] 1778347459.961893 write 0x1d2129 thru 0x1d2f4d (0xe25 bytes) 000316[1] 1778347459.973705 read 0x76d66e thru 0x76e8bc (0x124f bytes) 000317[1] 1778347459.979329 trunc from 0x946b1d to 0x616fb1 (0x32fb6b bytes) 000318[0] 1778347459.990782 trunc from 0x616fb2 to 0x7e353c (0x1cc58b bytes) 000320[0] 1778347460.006187 mapread 0x6dd654 thru 0x6df4d6 (0x1e83 bytes) 000321[1] 1778347460.014253 mapread 0x00fd01 thru 0x011476 (0x1776 bytes) 000322[1] 1778347460.024182 write 0x218b88 thru 0x21a92b (0x1da4 bytes) 000323[1] 1778347460.033245 read 0x2b9d09 thru 0x2ba317 (0x60f bytes) 000325[1] 1778347460.043283 mapread 0x1f9b0e thru 0x1fabe6 (0x10d9 bytes) 000327[0] 1778347460.055334 mapread 0x7cd30c thru 0x7cd890 (0x585 bytes) 000330[0] 1778347460.073221 write 0x6882b5 thru 0x68969f (0x13eb bytes) 000331[0] 1778347460.079893 mapread 0x55af2d thru 0x55cb63 (0x1c37 bytes) 000332[0] 1778347460.090870 read 0x3b1ad6 thru 0x3b25dd (0xb08 bytes) 000334[1] 1778347460.100608 mapwrite 0x15f0b9 thru 0x16075f (0x16a7 bytes) 000335[0] 1778347460.123444 read 0x5982eb thru 0x599ef2 (0x1c08 bytes) 000337[1] 1778347460.133364 mapread 0x350d82 thru 0x351ae2 (0xd61 bytes) 000339[1] 1778347460.145793 mapread 0x0234bd thru 0x025112 (0x1c56 bytes) 000342[1] 1778347460.164250 read 0x1e6676 thru 0x1e66d9 (0x64 bytes) 000344[0] 1778347460.173561 read 0x315750 thru 0x317106 (0x19b7 bytes) 000346[0] 1778347460.185898 read 0x09e2b3 thru 0x09f161 (0xeaf bytes) 000348[0] 1778347460.191451 mapread 0x74e72e thru 0x74e975 (0x248 bytes) 000350[0] 1778347460.197376 mapread 0x7cbdb0 thru 0x7cd17a (0x13cb bytes) 000352[0] 1778347460.205091 mapread 0x2e92ef thru 0x2e9fe1 (0xcf3 bytes) 000353[1] 1778347460.206861 trunc from 0x7e353d to 0x13eff1 (0x6a454b bytes) 000355[0] 1778347460.228310 trunc from 0x13eff2 to 0x13f2d5 (0x2e4 bytes) 000358[1] 1778347460.237484 read 0x0b47bc thru 0x0b51af (0x9f4 bytes) 000363[1] 1778347460.260953 mapwrite 0x055b53 thru 0x055c4b (0xf9 bytes) 000364[0] 1778347460.295270 read 0x119e2b thru 0x11a1b1 (0x387 bytes) 000366[0] 1778347460.315384 read 0x0be737 thru 0x0be817 (0xe1 bytes) 000369[0] 1778347460.323408 trunc from 0x13f2d6 to 0x629b9d (0x4ea8c8 bytes) 000372[1] 1778347460.339051 mapwrite 0x0676b3 thru 0x0689d7 (0x1325 bytes) 000374[1] 1778347460.381578 read 0x3bf8d4 thru 0x3c116c (0x1899 bytes) 000375[0] 1778347460.385465 mapwrite 0x52eea6 thru 0x52f6f8 (0x853 bytes) 000376[1] 1778347460.404309 mapread 0x1d7539 thru 0x1d8210 (0xcd8 bytes) 000378[1] 1778347460.413981 write 0x3228f1 thru 0x3230a9 (0x7b9 bytes) 000380[1] 1778347460.421379 write 0x3252b5 thru 0x326608 (0x1354 bytes) 000385[0] 1778347460.440219 trunc from 0x629b9e to 0x0efdc4 (0x539dd9 bytes) 000387[1] 1778347460.457966 read 0x06fdb4 thru 0x070eb0 (0x10fd bytes) 000388[0] 1778347460.471313 mapread 0x0402ec thru 0x04223e (0x1f53 bytes) 000389[1] 1778347460.483270 mapread 0x0d21a9 thru 0x0d2a96 (0x8ee bytes) 000390[1] 1778347460.489117 mapwrite 0x71a836 thru 0x71b00c (0x7d7 bytes) 000391[1] 1778347460.559701 mapread 0x1482f2 thru 0x1483e5 (0xf4 bytes) 000392[0] 1778347460.564302 read 0x14a72e thru 0x14bb6b (0x143e bytes) 000393[0] 1778347460.575102 write 0x4371bb thru 0x437cbd (0xb03 bytes) 000394[1] 1778347460.582481 mapwrite 0x57dd17 thru 0x57fba4 (0x1e8e bytes) 000396[0] 1778347460.608733 write 0x3968cb thru 0x396dea (0x520 bytes) 000397[1] 1778347460.616403 mapread 0x13d968 thru 0x13e1a1 (0x83a bytes) 000398[0] 1778347460.624432 mapread 0x2f2ecd thru 0x2f31a1 (0x2d5 bytes) 000399[1] 1778347460.629287 write 0x5b65ce thru 0x5b7b1b (0x154e bytes) 000401[0] 1778347460.635534 read 0x17051e thru 0x170cd3 (0x7b6 bytes) 000402[0] 1778347460.637473 mapwrite 0x8fcd12 thru 0x8fce9a (0x189 bytes) 000408[1] 1778347460.675703 trunc from 0x8fce9b to 0x186331 (0x776b69 bytes) 000410[1] 1778347460.690791 write 0x361c84 thru 0x361eec (0x269 bytes) 000412[0] 1778347460.692585 write 0x4ac028 thru 0x4ac4c9 (0x4a2 bytes) 000415[0] 1778347460.709154 write 0x1aa90d thru 0x1ab1a9 (0x89d bytes) 000416[0] 1778347460.715719 write 0x4e1457 thru 0x4e33ef (0x1f99 bytes) 000419[1] 1778347460.721157 mapwrite 0x9271f7 thru 0x928759 (0x1563 bytes) 000420[1] 1778347460.761908 write 0x079764 thru 0x07aa65 (0x1302 bytes) 000423[0] 1778347460.766000 mapwrite 0x9d4616 thru 0x9d4eb0 (0x89b bytes) 000425[1] 1778347460.792813 mapwrite 0x019367 thru 0x01ae7e (0x1b18 bytes) 000426[0] 1778347460.818179 mapread 0x5ef04c thru 0x5f0034 (0xfe9 bytes) 000429[0] 1778347460.834394 mapread 0x7bbf6c thru 0x7bc378 (0x40d bytes) 000430[1] 1778347460.843148 mapwrite 0x9e7c9c thru 0x9e81db (0x540 bytes) 000431[1] 1778347460.871318 write 0x171783 thru 0x1725af (0xe2d bytes) 000432[1] 1778347460.874523 mapwrite 0x706386 thru 0x706a0c (0x687 bytes) 000433[1] 1778347460.895296 mapwrite 0x858906 thru 0x859690 (0xd8b bytes) 000435[0] 1778347460.912717 trunc from 0x9e81dc to 0x340a69 (0x6a7772 bytes) 000436[0] 1778347460.924534 read 0x173551 thru 0x174709 (0x11b9 bytes) 000437[0] 1778347460.952648 mapwrite 0x31e42b thru 0x31e84a (0x420 bytes) 000438[1] 1778347460.965310 write 0x71c3a5 thru 0x71c7e3 (0x43f bytes) 000443[0] 1778347460.989042 mapread 0x08b352 thru 0x08cf46 (0x1bf5 bytes) 000444[0] 1778347460.995019 mapwrite 0x91ff64 thru 0x921aaf (0x1b4c bytes) 000445[0] 1778347461.058938 trunc from 0x921ab0 to 0x9a4d15 (0x83266 bytes) 000447[1] 1778347461.071883 mapread 0x3ff1ab thru 0x3ff454 (0x2aa bytes) 000448[1] 1778347461.087027 trunc from 0x9a4d16 to 0x7e3182 (0x1c1b93 bytes) 000450[1] 1778347461.100029 mapread 0x7af916 thru 0x7b0320 (0xa0b bytes) 000453[0] 1778347461.115751 mapwrite 0x9be39b thru 0x9beff5 (0xc5b bytes) 000455[1] 1778347461.148078 write 0x5049a1 thru 0x506055 (0x16b5 bytes) 000456[0] 1778347461.157221 mapwrite 0x674839 thru 0x675d62 (0x152a bytes) 000457[1] 1778347461.176188 read 0x0203af thru 0x02159a (0x11ec bytes) 000458[1] 1778347461.184127 mapread 0x3770fe thru 0x378f9a (0x1e9d bytes) 000460[1] 1778347461.192583 write 0x91c6a6 thru 0x91e5dd (0x1f38 bytes) 000463[0] 1778347461.206964 write 0x6fbd3c thru 0x6fcef4 (0x11b9 bytes) 000464[0] 1778347461.212489 write 0x3c4bea thru 0x3c69a1 (0x1db8 bytes) 000465[1] 1778347461.217922 write 0x22f27b thru 0x230324 (0x10aa bytes) 000470[0] 1778347461.238877 mapread 0x7564cf thru 0x756a4b (0x57d bytes) 000472[0] 1778347461.248236 trunc from 0x9beff6 to 0x858f08 (0x1660ed bytes) 000476[1] 1778347461.265273 mapread 0x09dc89 thru 0x09e125 (0x49d bytes) 000477[1] 1778347461.273395 mapwrite 0x71720e thru 0x71774d (0x540 bytes) 000479[1] 1778347461.292387 write 0x85bea4 thru 0x85d382 (0x14df bytes) 000480[0] 1778347461.298769 trunc from 0x85d383 to 0x41aa8c (0x4428f6 bytes) 000487[1] 1778347461.313952 mapread 0x1cba5a thru 0x1cd640 (0x1be7 bytes) 000488[0] 1778347461.324822 mapread 0x27abac thru 0x27b569 (0x9be bytes) 000492[0] 1778347461.335479 trunc from 0x41aa8d to 0x1ab60e (0x26f47e bytes) 000495[1] 1778347461.347282 write 0x394338 thru 0x395914 (0x15dd bytes) 000497[0] 1778347461.356783 read 0x273c43 thru 0x2740a6 (0x464 bytes) 000498[0] 1778347461.380043 read 0x029e1f thru 0x02a7d1 (0x9b3 bytes) 000499[0] 1778347461.380682 read 0x221501 thru 0x22227f (0xd7f bytes) All operations completed A-OK! PASS 16b (6s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 13:24:22 (1778347462) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (2s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 13:24:24 (1778347464) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0911075 s, 92.1 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0659993 s, 127 MB/s ldlm.namespaces.MGC192.168.204.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9e53829a2000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9e5383546800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9e53829a2000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9e5383546800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9e53829a2000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9e5383546800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.26761 s, 46.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 0.716713 s, 146 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 0.781202 s, 134 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.4055 s, 74.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.42525 s, 73.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.12237 s, 93.4 MB/s PASS 16d (11s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 13:24:35 (1778347475) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0852366 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296708 s, 35.3 MB/s PASS 16e (2s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 13:24:37 (1778347477) TIMEOUT PASS 16f (22s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 13:24:59 (1778347499) TIMEOUT PASS 16g (23s) == sanityn test 16h: mmap read after truncate file ======= 13:25:22 (1778347522) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (2s) == sanityn test 16i: read after truncate file ============ 13:25:24 (1778347524) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (2s) == sanityn test 16j: race dio with buffered i/o ========== 13:25:26 (1778347526) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.480144 s, 54.6 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.339869 s, 77.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.519165 s, 50.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.357676 s, 73.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.49154 s, 53.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.442534 s, 59.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.582291 s, 45.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.619365 s, 42.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.635337 s, 41.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.662863 s, 39.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.685431 s, 38.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.525004 s, 49.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.53502 s, 49.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.622215 s, 42.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.655153 s, 40.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.54706 s, 47.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.734909 s, 35.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.769475 s, 34.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.856607 s, 30.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (9s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 13:25:35 (1778347535) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x32cce75 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x1a7062c skipping zero size read truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x3c569db truncating to largest ever: 0x1a7062c truncating to largest ever: 0x32cce75 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 All operations completed A-OK! Sat May 9 13:25:42 EDT 2026 + fsx 2 done: rc=0 Sat May 9 13:25:42 EDT 2026 + fsx 3 done: rc=0 Sat May 9 13:25:42 EDT 2026 + fsx 4 done: rc=0 Sat May 9 13:25:42 EDT 2026 + fsx 5 done: rc=0 Sat May 9 13:25:42 EDT 2026 PASS 16k (10s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 13:25:45 (1778347545) fail_loc=0x8000030a PASS 17 (4s) == sanityn test 18: mmap sanity check =========================================================================================== 13:25:49 (1778347549) mmap test1: basic mmap operation (PASS, 0.026543s) mmap test2: MAP_PRIVATE not write back (PASS, 0.01934s) mmap test3: concurrent mmap ops on two nodes (PASS, 2.2243s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.1045s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.033569s) mmap test6: check mmap write/read content on two nodes (PASS, 0.03975s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 0.43175s) mmap test11: multi-thread shared mmap (PASS, 0.42241s) mmap test10: mtime not change for readonly mmap access (PASS, 1.0255s) PASS 18 (11s) == sanityn test 19: test concurrent uncached read races ========================================================================= 13:26:01 (1778347561) oleg435-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg435-client: oleg435-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (1s) == sanityn test 20: test extra readahead page left in cache ============================================================== 13:26:02 (1778347562) PASS 20 (2s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 13:26:04 (1778347564) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (2s) == sanityn test 23: others should see updated atime while another read============================================================== 13:26:06 (1778347566) now is 1778347567 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7598 new atime is 1778347628 PASS 23 (63s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 13:27:10 (1778347630) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 387 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 402 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 387 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 387 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 402 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 387 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 5.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 37.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537417 387 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118098 402 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234915 387 234528 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 387 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 402 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 387 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537417 387 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118098 402 117696 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre[OST:1] filesystem_summary: 234915 387 234528 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537417 387 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118098 402 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234915 387 234528 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 PASS 24a (2s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 13:27:12 (1778347632) PASS 24b (2s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 13:27:14 (1778347634) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (2s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 13:27:16 (1778347636) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (1s) == sanityn test 26a: allow mtime to get older ============ 13:27:17 (1778347637) utime: good mknod times 1778347636 <= 1778347637 <= 1778347637 for /mnt/lustre/f26a utime: good mknod times 1778347636 <= 1778347637 <= 1778347637 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (3s) == sanityn test 26b: sync mtime between ost and mds ====== 13:27:20 (1778347640) PASS 26b (4s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 13:27:24 (1778347644) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7598 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-05-09 13:27:24.000000000 -0400 Birth: 2026-05-09 13:27:24.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-05-09 13:27:24.000000000 -0400 Birth: 2026-05-09 13:27:24.000000000 -0400 PASS 26c (3s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 13:27:27 (1778347647) 0 PASS 30 (4s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 13:27:31 (1778347651) fail_loc=0x314 PASS 31a (3s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 13:27:34 (1778347654) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg435-server mds-ost sync done. sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.135@tcp:/lustre 7542784 39936 7496704 1% /mnt/lustre2 PASS 31b (18s) == sanityn test 31r: open-rename(replace) race =========== 13:27:53 (1778347673) -rw-r--r-- 1 root root 0 May 9 13:27 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 mv: failed to access '/mnt/lustre2/f31r.sanityn.target': Input/output error PASS 31r (5s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 13:27:58 (1778347678) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1207: 42994 Terminated tail -f $tfile PASS 31s (2s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 13:28:00 (1778347680) PASS 31t (3s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 13:28:04 (1778347684) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (1s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 13:28:05 (1778347685) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (1s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 13:28:06 (1778347686) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (1s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 13:28:07 (1778347687) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (1s) == sanityn test 34: no lock timeout under IO ============= 13:28:08 (1778347688) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg435-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9e53829a2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e53829a2000.ost_server_uuid in FULL state after 0 sec oleg435-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e53829a2000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e53829a2000.ost_server_uuid in IDLE state after 0 sec oleg435-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9e53829a2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e53829a2000.ost_server_uuid in FULL state after 0 sec oleg435-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e53829a2000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e53829a2000.ost_server_uuid in IDLE state after 0 sec oleg435-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9e53829a2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e53829a2000.ost_server_uuid in FULL state after 0 sec oleg435-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e53829a2000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e53829a2000.ost_server_uuid in IDLE state after 0 sec PASS 34 (91s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 13:29:39 (1778347779) fail_loc=0x80000317 Race attempt 0 Wait for 49034 49169 for 60 sec... total: 4000 open/close in 8.73 seconds: 458.24 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.204.135@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre 192.168.204.135@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre2 PASS 35 (65s) == sanityn test 36: handle ESTALE/open-unlink correctly == 13:30:44 (1778347844) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg435-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.367109 s, 143 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7598 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.34285 s, 153 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7598 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.3532 s, 148 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7598 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7497728) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.347048 s, 151 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7598 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7495680) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.408478 s, 128 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7598 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7495680) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.377839 s, 139 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7598 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7497728) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.396903 s, 132 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7598 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7495680) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.398475 s, 132 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7598 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7495680) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.4392 s, 119 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7598 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.41513 s, 126 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7598 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.362218 s, 145 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7598 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7497728) after_dd(7446528) after(7497728) PASS 36 (238s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 13:34:42 (1778348082) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7598 - create 8176 (time 1778348093.73 total 10.00 last 817.59) total: 10000 create in 12.43 seconds: 804.82 ops/second PASS 37 (16s) == sanityn test 39a: file mtime does not change after rename ========================================================== 13:34:58 (1778348098) repeat after cancel_lru_locks PASS 39a (3s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 13:35:01 (1778348101) repeat after cancel_lru_locks PASS 39b (3s) == sanityn test 39c: check truncate mtime update ================================================================================ 13:35:04 (1778348104) repeat after cancel_lru_locks PASS 39c (3s) == sanityn test 39d: sync write should update mtime ====== 13:35:07 (1778348107) fail_loc=0x411 fail_loc=0 PASS 39d (2s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 13:35:10 (1778348110) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (8s) == sanityn test 40c: pdirops: link and others ======================================================================== 13:35:18 (1778348118) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (8s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 13:35:26 (1778348126) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (8s) == sanityn test 40e: pdirops: rename and others ======================================================================== 13:35:34 (1778348134) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (7s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 13:35:42 (1778348142) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (5s) == sanityn test 41b: pdirops: create vs create ======================================================================== 13:35:47 (1778348147) open: File exists Conflict PASS 41b (5s) == sanityn test 41c: pdirops: create vs link ======================================================================== 13:35:52 (1778348152) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (6s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 13:35:58 (1778348158) Conflict PASS 41d (6s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 13:36:04 (1778348164) Conflict PASS 41e (5s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 13:36:10 (1778348170) Conflict PASS 41f (5s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 13:36:15 (1778348175) Conflict PASS 41g (5s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 13:36:21 (1778348181) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2318: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (6s) == sanityn test 41i: reint_open: create vs create ======== 13:36:26 (1778348186) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (724s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 13:48:30 (1778348910) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (6s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 13:48:36 (1778348916) open: File exists Conflict PASS 42b (5s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 13:48:42 (1778348922) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (6s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 13:48:47 (1778348927) Conflict PASS 42d (5s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 13:48:52 (1778348932) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (6s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 13:48:58 (1778348938) Conflict PASS 42f (5s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 13:49:03 (1778348943) Conflict PASS 42g (6s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 13:49:09 (1778348949) Conflict PASS 42h (6s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 13:49:15 (1778348955) PASS 43a (17s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 13:49:32 (1778348972) Conflict PASS 43b (5s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 13:49:37 (1778348977) Conflict PASS 43c (5s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 13:49:42 (1778348982) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (5s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 13:49:47 (1778348987) Conflict PASS 43e (5s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 13:49:52 (1778348992) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (5s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 13:49:57 (1778348997) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (5s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 13:50:02 (1778349002) Conflict PASS 43h (4s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 13:50:07 (1778349007) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (0s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 13:50:07 (1778349007) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists PASS 43j (44s) == sanityn test 43k: unlink vs create ==================== 13:50:51 (1778349051) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (463s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 13:58:34 (1778349514) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Sat May 9 13:58:37 EDT 2026 PASS 44a (5s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 13:58:39 (1778349519) open: File exists Conflict PASS 44b (6s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 13:58:45 (1778349525) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (6s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 13:58:51 (1778349531) Conflict PASS 44d (6s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 13:58:57 (1778349537) Conflict PASS 44e (6s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 13:59:03 (1778349543) Conflict PASS 44f (6s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 13:59:09 (1778349549) Conflict PASS 44g (5s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 13:59:15 (1778349555) Conflict PASS 44h (6s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 13:59:21 (1778349561) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (2s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 13:59:23 (1778349563) PASS 45a (34s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 13:59:57 (1778349597) Conflict PASS 45b (5s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 14:00:02 (1778349602) Conflict PASS 45c (6s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 14:00:08 (1778349608) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (5s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 14:00:13 (1778349613) Conflict PASS 45e (6s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 14:00:19 (1778349619) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (6s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 14:00:25 (1778349625) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (5s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 14:00:30 (1778349630) Conflict PASS 45h (5s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 14:00:35 (1778349635) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (1s) == sanityn test 45j: read vs rename ====================== 14:00:36 (1778349636) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (491s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 14:08:47 (1778350127) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (6s) == sanityn test 46b: pdirops: link vs create ======================================================================== 14:08:53 (1778350133) open: File exists Conflict PASS 46b (6s) == sanityn test 46c: pdirops: link vs link ======================================================================== 14:08:59 (1778350139) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (6s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 14:09:05 (1778350145) Conflict PASS 46d (6s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 14:09:11 (1778350151) Conflict PASS 46e (6s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 14:09:17 (1778350157) Conflict PASS 46f (6s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 14:09:23 (1778350163) Conflict PASS 46g (6s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 14:09:30 (1778350170) Conflict PASS 46h (6s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 14:09:36 (1778350176) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (1s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 14:09:38 (1778350178) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (1s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 14:09:39 (1778350179) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (1s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 14:09:41 (1778350181) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (1s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 14:09:42 (1778350182) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (2s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 14:09:44 (1778350184) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (1s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 14:09:45 (1778350185) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (2s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 14:09:47 (1778350187) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (1s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 14:09:48 (1778350188) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.00574022 s, 1.8 MB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (8s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 14:09:56 (1778350196) 0+1 records in 0+1 records out 158 bytes copied, 0.00621957 s, 25.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3581: kill: (217387) - No such process PASS 51a (5s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 14:10:01 (1778350201) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00614553 s, 167 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331429 s, 31.6 MB/s 3145728 PASS 51b (15s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 14:10:16 (1778350216) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00430186 s, 238 kB/s PASS 51c (7s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 14:10:23 (1778350223) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014145 s, 74.1 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (3s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 14:10:26 (1778350226) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2452 0x994 0x280000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2452 0x994 0x280000400 read lease(1) has applied. PASS 51e (5s) == sanityn test 54: rename locking ======================= 14:10:31 (1778350231) llite.lustre-ffff9e53829a2000.xattr_cache=1 llite.lustre-ffff9e5383546800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505011 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-09 14:10:31.000000000 -0400 Modify: 2026-05-09 14:10:31.000000000 -0400 Change: 2026-05-09 14:10:31.000000000 -0400 Birth: 2026-05-09 14:10:31.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505014 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-09 14:10:37.000000000 -0400 Modify: 2026-05-09 14:10:37.000000000 -0400 Change: 2026-05-09 14:10:37.000000000 -0400 Birth: 2026-05-09 14:10:36.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505017 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-09 14:10:42.000000000 -0400 Modify: 2026-05-09 14:10:42.000000000 -0400 Change: 2026-05-09 14:10:42.000000000 -0400 Birth: 2026-05-09 14:10:42.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505016 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-09 14:10:42.000000000 -0400 Modify: 2026-05-09 14:10:43.000000000 -0400 Change: 2026-05-09 14:10:43.000000000 -0400 Birth: 2026-05-09 14:10:42.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-09 14:10:48.000000000 -0400 Modify: 2026-05-09 14:10:48.000000000 -0400 Change: 2026-05-09 14:10:48.000000000 -0400 Birth: 2026-05-09 14:10:47.000000000 -0400 File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505019 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-09 14:10:48.000000000 -0400 Modify: 2026-05-09 14:10:48.000000000 -0400 Change: 2026-05-09 14:10:48.000000000 -0400 Birth: 2026-05-09 14:10:47.000000000 -0400 PASS 54 (24s) == sanityn test 55a: rename vs unlink target dir ========= 14:10:56 (1778350256) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (8s) == sanityn test 55b: rename vs unlink source dir ========= 14:11:04 (1778350264) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (9s) == sanityn test 55c: rename vs unlink orphan target dir == 14:11:13 (1778350273) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (13s) == sanityn test 55d: rename file vs link ================= 14:11:26 (1778350286) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (10s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 14:11:36 (1778350296) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (2s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 14:11:38 (1778350298) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (13s) == sanityn test 55g: rename: race with trylock =========== 14:11:51 (1778350311) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (15s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 14:12:06 (1778350326) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1778350327 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 1.60629 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (42s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 14:12:48 (1778350368) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1778350370 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 3.21248 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (43s) == sanityn test 60: Verify data_version behaviour ======== 14:13:32 (1778350412) PASS 60 (3s) == sanityn test 70a: cd directory && rm directory ======== 14:13:35 (1778350415) PASS 70a (3s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 14:13:38 (1778350418) PASS 70b (3s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 14:13:41 (1778350421) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00725293 s, 9.0 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (2s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 14:13:43 (1778350423) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00409135 s, 16.0 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (1s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 14:13:44 (1778350424) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (2s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 14:13:46 (1778350426) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0047525 s, 13.8 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (1s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 14:13:47 (1778350427) llite.lustre-ffff9e53829a2000.xattr_cache=1 llite.lustre-ffff9e5383546800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (3s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 14:13:50 (1778350430) llite.lustre-ffff9e53829a2000.xattr_cache=1 llite.lustre-ffff9e5383546800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (4s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 14:13:54 (1778350434) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00293591 s, 349 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00288692 s, 355 kB/s 230150: get lock2 230154: get lock1 230150: done 230150: get lock1 230154: done 230154: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 230154: cannot get lock2: Resource deadlock avoided 230154: exit rc=0 230150: done 230150: exit rc=0 PASS 74 (7s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 14:14:01 (1778350441) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0259534 s, 80.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139267 s, 75.3 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4264: 230736 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (7s) == sanityn test 76: Verify MDT open_files listing ======== 14:14:08 (1778350448) opening files: ........ closing files: ........ PASS 76 (24s) == sanityn test 77a: check FIFO NRS policy =============== 14:14:32 (1778350472) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.118525 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403738 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493493 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389461 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433128 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672432 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726613 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841796 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685853 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913049 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097125 s, 10.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109902 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935173 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835489 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851185 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116351 s, 9.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e53829a2000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9e53829a2000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10485 s, 10.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245996 s, 42.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214397 s, 48.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102356 s, 102 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016919 s, 62.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272649 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214421 s, 48.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193817 s, 54.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308348 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216824 s, 48.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204132 s, 51.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276262 s, 38.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166501 s, 63.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027967 s, 37.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235605 s, 44.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146483 s, 71.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e53829a2000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138459 s, 75.7 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (3s) == sanityn test 77b: check CRR-N NRS policy ============== 14:14:35 (1778350475) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.162806 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349008 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483259 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520354 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058397 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501691 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680252 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646405 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506631 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04996 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0974551 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099074 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877475 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111311 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11216 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126145 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131564 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241815 s, 43.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100404 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011411 s, 91.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170315 s, 61.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249458 s, 42.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153802 s, 68.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262595 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159177 s, 65.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245094 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186745 s, 56.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271329 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259466 s, 40.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224573 s, 46.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278178 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241057 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253901 s, 41.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e53829a2000.lock_unused_count=10 removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg435-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg435-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.13145 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255029 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595965 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619846 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646522 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607291 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553783 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066802 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789292 s, 13.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9e53829a2000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854728 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716739 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894496 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977795 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132128 s, 7.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00883072 s, 119 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133475 s, 7.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176818 s, 59.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019327 s, 54.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209693 s, 50.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263113 s, 39.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177763 s, 59.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024396 s, 43.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285827 s, 36.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115084 s, 91.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376332 s, 27.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149332 s, 70.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308701 s, 34.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300601 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160485 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04321 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374902 s, 28.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e53829a2000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150579 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00350597 s, 299 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (4s) == sanityn test 77c: check ORR NRS policy ================ 14:14:39 (1778350479) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.261918 s, 64.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343692 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501405 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556534 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523321 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750004 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611827 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543641 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092533 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0956608 s, 11.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896649 s, 11.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826276 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981344 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983937 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118331 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112582 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991939 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113923 s, 92.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195928 s, 53.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126915 s, 82.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129548 s, 80.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133924 s, 78.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018719 s, 56.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210848 s, 49.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183906 s, 57.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017563 s, 59.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229184 s, 45.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222177 s, 47.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224206 s, 46.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00552699 s, 190 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329197 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00374725 s, 280 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00364399 s, 288 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.161608 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027181 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451946 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579041 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652178 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658484 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490312 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567543 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652983 s, 16.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e53829a2000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9e53829a2000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941168 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105181 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0928031 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105681 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107998 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102194 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107407 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108607 s, 9.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e53829a2000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9e53829a2000.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172088 s, 60.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124971 s, 83.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161644 s, 64.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172727 s, 60.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244894 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00704869 s, 149 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216343 s, 48.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018864 s, 55.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175247 s, 59.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00380073 s, 276 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019673 s, 53.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238571 s, 44.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022884 s, 45.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0098433 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00449641 s, 233 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00184107 s, 570 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg435-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg435-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.196416 s, 85.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325482 s, 32.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393785 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442631 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850061 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814072 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697874 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910671 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973406 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949612 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823761 s, 12.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821401 s, 12.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e53829a2000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9e53829a2000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092915 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907018 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112935 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869734 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986036 s, 10.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017475 s, 60.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178882 s, 58.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243695 s, 43.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192818 s, 54.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188868 s, 55.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020519 s, 51.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252713 s, 41.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244191 s, 42.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218758 s, 47.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254308 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287734 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00846633 s, 124 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00792481 s, 132 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258884 s, 40.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e53829a2000.lock_unused_count=1 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270508 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138051 s, 76.0 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (6s) == sanityn test 77d: check TRR nrs policy ================ 14:14:45 (1778350485) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.20403 s, 82.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187977 s, 55.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576593 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596408 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737544 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837912 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941928 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905504 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620391 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955806 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726763 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064846 s, 16.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9e53829a2000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861957 s, 12.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953563 s, 11.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993221 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100637 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083591 s, 12.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204586 s, 51.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219518 s, 47.8 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff9e53829a2000.lock_unused_count=1 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195844 s, 53.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00728092 s, 144 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198317 s, 52.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181417 s, 57.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248887 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215032 s, 48.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0043741 s, 240 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00658912 s, 159 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180367 s, 58.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229224 s, 45.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00467572 s, 224 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022632 s, 46.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00340672 s, 308 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286295 s, 36.6 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg435-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg435-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.194601 s, 86.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468257 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487491 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043284 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395401 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656576 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513229 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826432 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103084 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106378 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589163 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911667 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105399 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573187 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102214 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101386 s, 10.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e53829a2000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9e53829a2000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10803 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169529 s, 61.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167985 s, 62.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143356 s, 73.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232676 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131247 s, 79.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254861 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00947974 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257903 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188157 s, 55.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00638259 s, 164 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227499 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00865303 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283658 s, 37.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248606 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00253458 s, 414 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00203421 s, 515 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg435-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg435-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.180022 s, 93.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544158 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548295 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046982 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797032 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754508 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10906 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119739 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118021 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138962 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128386 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15424 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145657 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128386 s, 8.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e53829a2000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9e53829a2000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160792 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129859 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171897 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142894 s, 73.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276678 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218158 s, 48.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251509 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270863 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344829 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325384 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430019 s, 24.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326474 s, 32.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364233 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348777 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424973 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011421 s, 91.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510985 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175835 s, 59.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497485 s, 21.1 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (6s) == sanityn test 77e: check TBF NID nrs policy ============ 14:14:51 (1778350491) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.204.35@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.163233 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035368 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416838 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497506 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431375 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296338 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048492 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516419 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577167 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376771 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804102 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873523 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901771 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787575 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087673 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925808 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091865 s, 11.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e53829a2000.lock_unused_count=3 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297361 s, 35.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352678 s, 29.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485889 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449966 s, 23.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302734 s, 34.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469601 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05269 s, 19.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031692 s, 33.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406599 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308124 s, 34.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517791 s, 20.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442974 s, 23.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453505 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360482 s, 29.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395326 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037731 s, 27.8 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.135479 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215045 s, 48.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481758 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551738 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565338 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539554 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045575 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394816 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607911 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840248 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881773 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970994 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921667 s, 11.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111014 s, 9.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106145 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10671 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111605 s, 9.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e53829a2000.lock_unused_count=4 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804115 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759868 s, 13.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077501 s, 13.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640255 s, 16.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070585 s, 14.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828639 s, 12.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688831 s, 15.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770157 s, 13.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706267 s, 14.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680169 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076235 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692313 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775585 s, 13.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787691 s, 13.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769673 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765971 s, 13.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.192002 s, 87.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452641 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489715 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462345 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625469 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539648 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639633 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992161 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118172 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105336 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133608 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112777 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155677 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151736 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13558 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153418 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139245 s, 75.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210451 s, 49.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257031 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00765602 s, 137 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300541 s, 34.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168567 s, 62.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e53829a2000.lock_unused_count=8 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031405 s, 33.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00934124 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0089227 s, 118 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253144 s, 41.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326852 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342816 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400546 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381339 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367787 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154181 s, 68.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177749 s, 5.9 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.169799 s, 98.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321009 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604321 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819003 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110736 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13502 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112863 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121181 s, 8.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135432 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126995 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146659 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152632 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147016 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162287 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154941 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133189 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171597 s, 6.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e53829a2000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9e53829a2000.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123637 s, 84.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173784 s, 60.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324625 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224913 s, 46.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103174 s, 102 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171771 s, 61.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305655 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293961 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216607 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00522548 s, 201 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330164 s, 31.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343864 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153904 s, 68.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310858 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109834 s, 95.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184194 s, 56.9 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg435-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg435-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (10s) == sanityn test 77f: check TBF JobID nrs policy ========== 14:15:01 (1778350501) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg435-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.199294 s, 84.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351161 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085238 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755093 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853231 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968324 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741874 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766371 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739556 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789572 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126535 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123981 s, 8.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158782 s, 6.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15055 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141363 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128853 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132545 s, 7.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e53829a2000.lock_unused_count=7 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233533 s, 44.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265626 s, 39.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170667 s, 61.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180588 s, 58.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234385 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150687 s, 69.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208837 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151674 s, 69.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207169 s, 50.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174489 s, 60.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192803 s, 54.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283703 s, 37.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181318 s, 57.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017501 s, 59.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146423 s, 71.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00621017 s, 169 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg435-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.181825 s, 92.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367563 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370864 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725717 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880905 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081715 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715404 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873968 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913581 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111147 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10055 s, 10.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962341 s, 10.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104315 s, 10.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961611 s, 10.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115771 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091081 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112036 s, 9.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e53829a2000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9e53829a2000.lock_unused_count=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150608 s, 69.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183287 s, 57.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156775 s, 66.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151743 s, 69.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179054 s, 58.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e53829a2000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152082 s, 68.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103137 s, 102 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00430196 s, 244 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235139 s, 44.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226716 s, 46.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00536149 s, 196 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012924 s, 81.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173383 s, 60.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00536263 s, 196 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232651 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00878967 s, 119 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg435-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.251553 s, 66.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255123 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674197 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050811 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745744 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755244 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519766 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884706 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773803 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778434 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668061 s, 15.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9e53829a2000.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922427 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204716 s, 51.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185324 s, 56.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158958 s, 66.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132699 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173384 s, 60.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147246 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138938 s, 7.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136983 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243439 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169101 s, 62.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130392 s, 8.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149205 s, 70.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00914868 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00354222 s, 296 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178745 s, 58.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190827 s, 54.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324108 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00948306 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00360303 s, 291 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151648 s, 69.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160407 s, 65.4 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg435-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.207137 s, 81.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286906 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437342 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420878 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100399 s, 10.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11526 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122382 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12808 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139487 s, 7.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133954 s, 7.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145712 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146538 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158364 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129696 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142048 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147804 s, 7.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e53829a2000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9e53829a2000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159578 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493639 s, 21.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333943 s, 31.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350984 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368426 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323882 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405734 s, 25.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341047 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331347 s, 31.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284453 s, 36.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035394 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388353 s, 27.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367539 s, 28.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357907 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318657 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060929 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361907 s, 29.0 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (9s) == sanityn test 77g: Change TBF type directly ============ 14:15:10 (1778350510) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (4s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 14:15:14 (1778350514) oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg435-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg435-client: oleg435-server: ssh exited with exit code 2 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg435-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg435-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg435-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 PASS 77h (5s) == sanityn test 77i: Change rank of TBF rule ============= 14:15:19 (1778350519) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (9s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 14:15:28 (1778350528) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.170369 s, 98.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03135 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250086 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628936 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586578 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926211 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907217 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863302 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111551 s, 9.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117202 s, 8.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115674 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104297 s, 10.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113509 s, 9.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10198 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108125 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10511 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122366 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00668984 s, 157 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223651 s, 46.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255444 s, 41.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302863 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027707 s, 37.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029636 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352791 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246442 s, 42.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278671 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110619 s, 94.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038191 s, 27.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282738 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00766474 s, 137 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00439255 s, 239 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109482 s, 95.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00412789 s, 254 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.86235 s, 21.6 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.3033 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (55s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 14:16:23 (1778350583) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg435-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg435-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg435-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg435-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg435-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.2148 s, 13.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644567 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877752 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066798 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079731 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097556 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817433 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960433 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757001 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149871 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150038 s, 7.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163467 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165088 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132484 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142761 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194342 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176901 s, 5.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125823 s, 83.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148055 s, 70.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214849 s, 48.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226555 s, 46.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169881 s, 61.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251074 s, 41.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00362997 s, 289 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260317 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226468 s, 46.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216498 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00510005 s, 206 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214192 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242265 s, 43.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00576051 s, 182 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234436 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00487157 s, 215 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg435-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg435-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6987 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9014 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg435-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg435-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg435-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg435-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg435-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.22491 s, 13.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929512 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931408 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081358 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892571 s, 11.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076607 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817437 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082801 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846235 s, 12.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135672 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135679 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138248 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116608 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14764 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131056 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119503 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144229 s, 7.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023234 s, 45.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246506 s, 42.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216934 s, 48.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195153 s, 53.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284552 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193145 s, 54.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178437 s, 58.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279611 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170877 s, 61.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270851 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239572 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0062231 s, 168 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274998 s, 38.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248194 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252986 s, 41.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103095 s, 102 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6673 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9178 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (134s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 14:18:37 (1778350717) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg435-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg435-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg435-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg435-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg435-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.22027 s, 13.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735699 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697707 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706433 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739667 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820672 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778876 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097792 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731425 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133527 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133573 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140218 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143709 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148699 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156353 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154813 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158972 s, 6.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152056 s, 69.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232675 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220561 s, 47.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229274 s, 45.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251358 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254513 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223124 s, 47.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195801 s, 53.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177928 s, 58.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198107 s, 52.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209217 s, 50.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021813 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219823 s, 47.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030007 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250659 s, 41.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265105 s, 39.6 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg435-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg435-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5591 s, 5.4 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.935 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg435-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg435-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg435-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg435-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg435-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.21249 s, 13.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818224 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111966 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156267 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156392 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144884 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152552 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138663 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154122 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159002 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158398 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143765 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166931 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172347 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180715 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192643 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165875 s, 6.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282577 s, 37.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344633 s, 30.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305091 s, 34.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414138 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304901 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396086 s, 26.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159227 s, 65.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192243 s, 54.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230289 s, 45.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440335 s, 23.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401068 s, 26.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024296 s, 43.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373197 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316688 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421251 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176431 s, 59.4 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6218 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9439 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (139s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 14:20:56 (1778350856) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg435-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.282577 s, 59.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616931 s, 17.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617534 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732235 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783025 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673609 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758489 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987432 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07357 s, 14.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925865 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883696 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103261 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885801 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953645 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839514 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856092 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993525 s, 10.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154033 s, 68.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216888 s, 48.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207476 s, 50.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032008 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158508 s, 66.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244701 s, 42.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229985 s, 45.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019522 s, 53.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253631 s, 41.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243158 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029958 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00848602 s, 124 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.003856 s, 272 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107029 s, 98.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301232 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00388494 s, 270 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg435-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg435-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.86208 s, 21.6 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.71548 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.35@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.35@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.224705 s, 74.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300968 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074383 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744588 s, 14.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927207 s, 11.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775777 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894408 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753727 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966432 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907306 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856007 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139511 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145718 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252929 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250249 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293848 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297393 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202575 s, 51.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151504 s, 69.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023289 s, 45.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215109 s, 48.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00635174 s, 165 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242632 s, 43.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146081 s, 71.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118464 s, 88.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225685 s, 46.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111322 s, 94.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157717 s, 66.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00862001 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121999 s, 85.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161311 s, 65.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108957 s, 96.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304498 s, 34.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.83307 s, 21.7 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.63836 s, 10.9 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.35@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg435-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.222957 s, 75.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435871 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608139 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622197 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695754 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652682 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060454 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935578 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723527 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826713 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754547 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615723 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785575 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856993 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816015 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915701 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841959 s, 12.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164225 s, 63.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234553 s, 44.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217766 s, 48.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00990237 s, 106 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229259 s, 45.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224397 s, 46.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256985 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00790796 s, 133 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232962 s, 45.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240359 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227866 s, 46.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00387624 s, 271 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228622 s, 45.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281013 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253649 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00172783 s, 607 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.86343 s, 21.6 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.86185 s, 21.6 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg435-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.424421 s, 39.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028043 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043692 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720989 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784023 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907333 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10393 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922367 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951478 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106832 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110559 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105927 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159314 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137539 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165405 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178271 s, 5.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152388 s, 68.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192931 s, 54.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179119 s, 58.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185587 s, 56.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197778 s, 53.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021252 s, 49.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240839 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238376 s, 44.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246832 s, 42.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256867 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260569 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266378 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267276 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288937 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481706 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521174 s, 20.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg435-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg435-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.6847 s, 10.8 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.72177 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.85207 s, 21.6 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.67478 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg435-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.831051 s, 20.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461397 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874001 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105067 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901807 s, 11.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119357 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101284 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105612 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102967 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144836 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174192 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192543 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212594 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212771 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204908 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200705 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211161 s, 5.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202928 s, 51.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270853 s, 38.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351778 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021467 s, 48.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252876 s, 41.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318617 s, 32.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324823 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309788 s, 33.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347562 s, 30.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152641 s, 68.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137992 s, 76.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213992 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332517 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00901042 s, 116 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00871198 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102234 s, 103 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg435-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg435-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.2742 s, 5.4 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4152 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg435-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.251498 s, 66.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031678 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268669 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09481 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095028 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863018 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926602 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103638 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102012 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942733 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102357 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915131 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102871 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116276 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905768 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110545 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129447 s, 8.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017661 s, 59.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251534 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205433 s, 51.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179986 s, 58.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194526 s, 53.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196809 s, 53.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177174 s, 59.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193599 s, 54.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206162 s, 50.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253115 s, 41.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178835 s, 58.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206976 s, 50.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256288 s, 40.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249947 s, 42.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256368 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00191421 s, 548 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.85811 s, 21.6 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.71442 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg435-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.430553 s, 39.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113347 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116603 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143045 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163583 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131814 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148695 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13592 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148876 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173829 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143763 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147197 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149608 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160768 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153831 s, 6.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158942 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170367 s, 6.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022744 s, 46.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330389 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235107 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205931 s, 50.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330809 s, 31.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011712 s, 89.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310716 s, 33.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332789 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319527 s, 32.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323612 s, 32.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328669 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133247 s, 78.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038198 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00587615 s, 178 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00790472 s, 133 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256445 s, 40.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg435-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg435-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.63356 s, 10.9 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.71283 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.68145 s, 10.8 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.62552 s, 10.9 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (369s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 14:27:05 (1778351225) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg435-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.267459 s, 62.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554602 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922103 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806047 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104796 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11891 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124524 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123154 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124735 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140584 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135323 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145068 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127854 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160815 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142239 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166962 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142856 s, 7.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023963 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267336 s, 39.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134687 s, 77.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00975747 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106159 s, 98.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219707 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147986 s, 70.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00755108 s, 139 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035215 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459347 s, 22.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219614 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00329577 s, 318 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167478 s, 62.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178964 s, 58.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134028 s, 78.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105263 s, 99.6 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.86537 s, 21.6 MB/s Write runtime is 5 s, speed is 20.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7154 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (42s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 14:27:47 (1778351267) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.35@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.35@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.233931 s, 71.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358284 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448529 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665956 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100448 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105605 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103774 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123023 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131439 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139052 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14581 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246154 s, 4.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240782 s, 4.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257142 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2893 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348152 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313396 s, 3.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317235 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338707 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285945 s, 36.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024205 s, 43.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301975 s, 34.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156375 s, 67.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292964 s, 35.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199631 s, 52.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302559 s, 34.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102236 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427457 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196128 s, 53.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403061 s, 26.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135983 s, 77.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142895 s, 73.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124118 s, 84.5 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.88577 s, 21.5 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.64098 s, 10.9 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (41s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 14:28:28 (1778351308) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.35@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg435-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.2245 s, 74.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256422 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823207 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917066 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100344 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964407 s, 10.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093128 s, 11.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983017 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114591 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109107 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964472 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865305 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130247 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106074 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111447 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108976 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102126 s, 10.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245778 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160234 s, 65.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02661 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214273 s, 48.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175077 s, 59.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278483 s, 37.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00374376 s, 280 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120513 s, 87.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368622 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333515 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365424 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00328697 s, 319 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195821 s, 53.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342844 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00495662 s, 212 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00449353 s, 233 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg435-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg435-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.84374 s, 21.6 MB/s Write runtime is 5 s, speed is 20.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.91181 s, 21.3 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (36s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 14:29:04 (1778351344) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg435-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.448425 s, 37.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293201 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539706 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176905 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148256 s, 7.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154824 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148714 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152018 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188475 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151224 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163515 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175935 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176644 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168923 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171414 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171287 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185254 s, 5.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191787 s, 54.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301343 s, 34.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228255 s, 45.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234301 s, 44.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248474 s, 42.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130443 s, 80.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02181 s, 48.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154779 s, 67.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184269 s, 56.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0078964 s, 133 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356181 s, 29.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127419 s, 82.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00760969 s, 138 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024619 s, 42.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129595 s, 80.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0060647 s, 173 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg435-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg435-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7288 s, 10.8 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.71053 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.83059 s, 21.7 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.6114 s, 10.9 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (81s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 14:30:25 (1778351425) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg435-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.833477 s, 20.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398244 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787291 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885112 s, 11.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0928968 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877797 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983658 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107335 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931016 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13069 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175145 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156072 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186354 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164602 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19804 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180289 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169871 s, 6.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164674 s, 63.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147582 s, 71.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252691 s, 41.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166064 s, 63.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163779 s, 64.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286489 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231967 s, 45.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196579 s, 53.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00733441 s, 143 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223971 s, 46.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136208 s, 77.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227529 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00561151 s, 187 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147781 s, 71.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223261 s, 47.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00167318 s, 627 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg435-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg435-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.3895 s, 5.4 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8759 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (67s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 14:31:32 (1778351492) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg435-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.234122 s, 71.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240984 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348736 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730898 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552843 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888811 s, 11.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108649 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994458 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105651 s, 9.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107559 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101934 s, 10.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977177 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121315 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124491 s, 8.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112269 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111035 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105008 s, 10.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159154 s, 65.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172713 s, 60.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237014 s, 44.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153206 s, 68.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018287 s, 57.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115283 s, 91.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00460323 s, 228 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176532 s, 59.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00982933 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00403377 s, 260 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00324257 s, 323 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00899156 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025249 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270958 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00247006 s, 425 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00252075 s, 416 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg435-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg435-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.8663 s, 21.5 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.71334 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg435-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.453241 s, 37.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784634 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689104 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654907 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115672 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102786 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124353 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143903 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128983 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125869 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169356 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170265 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126015 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304686 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325939 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.41841 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.437283 s, 2.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213586 s, 49.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311945 s, 33.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212761 s, 49.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344975 s, 30.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159998 s, 65.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225539 s, 46.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135812 s, 77.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184398 s, 56.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013518 s, 77.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031323 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015108 s, 69.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203671 s, 51.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037743 s, 27.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373368 s, 28.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029825 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129593 s, 80.9 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg435-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg435-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.65604 s, 10.9 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.71375 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.72421 s, 10.8 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.69676 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (122s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 14:33:34 (1778351614) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg435-client.virtnet /mnt/lustre (opts:) Stopping client oleg435-client.virtnet /mnt/lustre opts: Stopping clients: oleg435-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg435-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg435-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg435-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg435-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) mqueue on /dev/mqueue type mqueue (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg435-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.35,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg435-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.35,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg435-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.35,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/63484/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.35,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg435-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg435-server' oleg435-server: oleg435-server.virtnet: executing load_modules_local oleg435-server: Loading modules from /home/green/git/lustre-release/lustre oleg435-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg435-server: detected 4 online CPUs by sysfs oleg435-server: MODOPTS_LIBCFS= oleg435-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg435-server: oleg435-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg435-server: oleg435-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg435-client.virtnet: -o user_xattr,flock 192.168.204.135@tcp:/lustre /mnt/lustre Starting client oleg435-client.virtnet: -o user_xattr,flock 192.168.204.135@tcp:/lustre /mnt/lustre Started clients oleg435-client.virtnet: 192.168.204.135@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Starting client: oleg435-client.virtnet: -o user_xattr,flock 192.168.204.135@tcp:/lustre /mnt/lustre2 Starting client oleg435-client.virtnet: -o user_xattr,flock 192.168.204.135@tcp:/lustre /mnt/lustre2 Started clients oleg435-client.virtnet: 192.168.204.135@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9e53899a0000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9e5389b35000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e53899a0000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e5389b35000.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.228614 s, 73.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019642 s, 53.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792292 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708797 s, 14.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774274 s, 13.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725589 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891833 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709344 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633095 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722213 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747872 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717337 s, 14.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249898 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259128 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272833 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27779 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300474 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179624 s, 58.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164962 s, 63.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266868 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282862 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217536 s, 48.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230171 s, 45.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278596 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00440353 s, 238 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019333 s, 54.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308045 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121405 s, 86.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390076 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00323341 s, 324 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262912 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265172 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00203397 s, 516 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg435-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg435-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.87768 s, 21.5 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.89554 s, 21.4 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.264383 s, 63.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370924 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270184 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113111 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13553 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132141 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115837 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125162 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129481 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132526 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144782 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132951 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146045 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190558 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207043 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235153 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224406 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166154 s, 63.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197581 s, 53.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177226 s, 59.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197434 s, 53.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214967 s, 48.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202186 s, 51.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185762 s, 56.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245321 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00572529 s, 183 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162953 s, 64.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00987939 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174845 s, 60.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00735519 s, 143 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00997256 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00862746 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157996 s, 66.4 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg435-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg435-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.85951 s, 21.6 MB/s Write runtime is 5 s, speed is 20.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.68541 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (142s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 14:35:56 (1778351756) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg435-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (8s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 14:36:04 (1778351764) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (4s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 14:36:08 (1778351768) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg435-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.56713 s, 294 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg435-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0155 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (52s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 14:37:00 (1778351820) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.182311 s, 92.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260977 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066485 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612732 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515371 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575904 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692968 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677097 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737186 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874189 s, 12.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e53899a0000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104772 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132392 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124322 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146226 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232604 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272044 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134374 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281236 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286212 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147144 s, 71.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364769 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171423 s, 61.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404921 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421627 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351693 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126884 s, 82.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9e53899a0000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15825 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400675 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498815 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053896 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173671 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0036436 s, 288 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00325463 s, 322 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg435-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg435-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.87592 s, 21.5 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.90578 s, 21.4 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.214275 s, 78.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339651 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483129 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666896 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951377 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865209 s, 12.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908999 s, 11.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925784 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937889 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978644 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842744 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893291 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873455 s, 12.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892994 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103106 s, 10.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943727 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092096 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00799359 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025491 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151228 s, 69.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244724 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232643 s, 45.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330239 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238155 s, 44.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191531 s, 54.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261497 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00514749 s, 204 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266441 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00457696 s, 229 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00408565 s, 257 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0047863 s, 219 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00401028 s, 261 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00309887 s, 338 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg435-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg435-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.84336 s, 21.6 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.90131 s, 21.4 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (68s) == sanityn test 77o: Changing rank should not panic ====== 14:38:08 (1778351888) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (5s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 14:38:13 (1778351893) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (48s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 14:39:01 (1778351941) mds.MDS.mdt.nrs_policies=tbf oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg435-client: oleg435-server: ssh exited with exit code 36 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (14s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 14:39:15 (1778351955) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 271554 kill md_thread xargs PASS 77r (40s) == sanityn test 77s: Check TBF LRU shrinker ============== 14:39:55 (1778351995) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 10 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 10 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (17s) == sanityn test 77t: check TBF minrate parsing and output ========================================================== 14:40:12 (1778352012) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start fio jobid={fio.*} minrate=5 rate=20 realtime=1 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=change fio rate=4: Invalid argument oleg435-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='change fio rate=4': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=change fio minrate=10 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=change fio minrate=21: Invalid argument oleg435-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='change fio minrate=21': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=change fio rate=30 ost.OSS.ost_io.nrs_tbf_rule=change fio minrate=0 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start bad jobid={bad.*} minrate=31 rate=30: Invalid argument oleg435-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start bad jobid={bad.*} minrate=31 rate=30': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop fio ost.OSS.ost_io.nrs_policies=fifo PASS 77t (13s) == sanityn test 77u: check TBF minrate on NID and generic rules ========================================================== 14:40:25 (1778352025) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start nid_min nid={0@lo 192.168.204.35@tcp} minrate=5 rate=20 ost.OSS.ost_io.nrs_tbf_rule=stop nid_min ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start combo_min jobid={dd.500}&opcode={ost_write} minrate=5 rate=20 ost.OSS.ost_io.nrs_tbf_rule=change combo_min minrate=0 ost.OSS.ost_io.nrs_tbf_rule=stop nid_min ost.OSS.ost_io.nrs_tbf_rule=stop combo_min ost.OSS.ost_io.nrs_policies=fifo PASS 77u (11s) == sanityn test 77v: verify minimum RPC rate under congestion ========================================================== 14:40:36 (1778352036) SKIP: sanityn test_77v need OSTs and MDTs on separate nodes SKIP 77v (1s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 14:40:37 (1778352037) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (4s) == sanityn test 79: xattr: intent error ================== 14:40:41 (1778352041) File: /mnt/lustre/d79.sanityn Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826936025 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-09 14:40:42.000000000 -0400 Modify: 2026-05-09 14:40:42.000000000 -0400 Change: 2026-05-09 14:40:42.000000000 -0400 Birth: 2026-05-09 14:40:42.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (14s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 14:40:55 (1778352055) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (1s) == sanityn test 80b: Accessing directory during migration ========================================================== 14:40:56 (1778352056) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (2s) == sanityn test 81a: rename and stat under striped directory ========================================================== 14:40:58 (1778352058) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (1s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 14:40:59 (1778352059) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (1s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 14:41:00 (1778352060) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (2s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 14:41:02 (1778352062) - open/close 2433 (time 1778352074.42 total 10.00 last 243.24) total: 2500 open/close in 10.29 seconds: 242.96 ops/second snapshot_time 1778352096.510165821 secs.nsecs start_time 1778352062.900235924 secs.nsecs elapsed_time 33.609929897 secs.nsecs open 2505 samples [usecs] 117 10987 1600334 1226179962 close 2504 samples [usecs] 70 4031 452882 123041084 mknod 2501 samples [usecs] 229 10878 1419842 991923582 unlink 2500 samples [usecs] 372 37014 2841712 13145402618 mkdir 2 samples [usecs] 975 1233 2208 2470914 rmdir 2 samples [usecs] 3141 8149 11290 76272082 rename 2500 samples [usecs] 299 17553 3572908 8145667544 getattr 3761 samples [usecs] 7 8099 98687 106887753 parallel_rename_file 2500 samples [usecs] 299 17553 3572908 8145667544 crossdir_rename 2500 samples [usecs] 299 17553 3572908 8145667544 PASS 81d (37s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 14:41:39 (1778352099) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7598 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7598 PASS 82 (3s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 14:41:42 (1778352102) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (1s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 14:41:43 (1778352103) fail_loc=0x8000060b PASS 84 (9s) == sanityn test 85: Lustre API root cache race =========== 14:41:52 (1778352112) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1778352113 Finishing test test1 at 1778352113 PASS 85 (3s) == sanityn test 90: open/create and unlink striped directory ========================================================== 14:41:55 (1778352115) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (2s) == sanityn test 91: chmod and unlink striped directory === 14:41:57 (1778352117) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (1s) == sanityn test 92: create remote directory under orphan directory ========================================================== 14:41:58 (1778352118) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (1s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 14:41:59 (1778352119) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6189 0x182d 0x240000400 1 6151 0x1807 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6152 0x1808 0x280000400 0 6190 0x182e 0x240000400 rc1=0 and rc2=0 PASS 93 (10s) == sanityn test 94: signal vs CP callback race =========== 14:42:09 (1778352129) osc.lustre-OST0000-osc-ffff9e53899a0000.idle_timeout=0 osc.lustre-OST0000-osc-ffff9e5389b35000.idle_timeout=0 osc.lustre-OST0001-osc-ffff9e53899a0000.idle_timeout=0 osc.lustre-OST0001-osc-ffff9e5389b35000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0760711 s, 108 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.09381 s, 1.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6558: 292838 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0231834 s, 177 kB/s osc.lustre-OST0000-osc-ffff9e53899a0000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9e5389b35000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e53899a0000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e5389b35000.idle_timeout=debug PASS 94 (11s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 14:42:20 (1778352140) llite.lustre-ffff9e53899a0000.fast_read=0 llite.lustre-ffff9e5389b35000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0252523 s, 83.0 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff9e53899a0000.fast_read=1 llite.lustre-ffff9e5389b35000.fast_read=1 PASS 95a (16s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 14:42:36 (1778352156) llite.lustre-ffff9e53899a0000.fast_read=0 llite.lustre-ffff9e5389b35000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00197838 s, 6.2 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff9e53899a0000.fast_read=1 llite.lustre-ffff9e5389b35000.fast_read=1 PASS 95b (9s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 14:42:45 (1778352165) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (1s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 14:42:46 (1778352166) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0028945 s, 1.4 MB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (3s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 14:42:50 (1778352170) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0668642 s, 31.4 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (3s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 14:42:53 (1778352173) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0630831 s, 33.2 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (3s) == sanityn test 100e: DoM: read on open and file size ==== 14:42:56 (1778352176) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00255684 s, 4.7 MB/s PASS 100e (3s) == sanityn test 101a: Discard DoM data on unlink ========= 14:42:59 (1778352179) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00240949 s, 1.7 MB/s PASS 101a (3s) == sanityn test 101b: Discard DoM data on rename ========= 14:43:02 (1778352182) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00276507 s, 1.5 MB/s PASS 101b (3s) == sanityn test 101c: Discard DoM data on close-unlink === 14:43:05 (1778352185) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00341167 s, 1.2 MB/s PASS 101c (4s) SKIP: sanityn test_102 skipping ALWAYS excluded test 102 == sanityn test 103: Test size correctness with lockahead ========================================================== 14:43:10 (1778352190) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1778352191 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1778352194 Finishing test test23 at 1778352194 PASS 103 (8s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 14:43:18 (1778352198) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (2s) == sanityn test 105: Glimpse and lock cancel race ======== 14:43:20 (1778352200) fail_loc=0x416 PASS 105 (24s) == sanityn test 106a: Verify the btime via statx() ======= 14:43:44 (1778352224) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (1s) == sanityn test 106b: Glimpse RPCs test for statx ======== 14:43:45 (1778352225) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00504755 s, 203 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938560 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-09 14:43:46.000000000 -0400 Modify: 2026-05-09 14:43:46.000000000 -0400 Change: 2026-05-09 14:43:46.000000000 -0400 Birth: 2026-05-09 14:43:46.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938560 -rw-r--r-- 1 0 0 1778352226 1778352226 1778352226 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938560 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-09 14:43:46.000000000 -0400 Modify: 2026-05-09 14:43:46.000000000 -0400 Change: 2026-05-09 14:43:46.000000000 -0400 Birth: 2026-05-09 14:43:46.000000000 -0400 1778352226 1024 1 PASS 106b (4s) == sanityn test 106c: Verify statx attributes mask ======= 14:43:49 (1778352229) PASS 106c (3s) == sanityn test 107a: Basic grouplock conflict =========== 14:43:52 (1778352232) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.141588 s, 74.1 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6198 0x1836 0x240000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7598 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7598 First grouplock blocks second one PASS 107a (6s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 14:43:58 (1778352238) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0975957 s, 107 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6159 0x180f 0x280000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7598 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (9s) == sanityn test 108a: lseek: parallel updates ============ 14:44:07 (1778352247) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00214745 s, 4.7 MB/s llite.lustre-ffff9e53899a0000.hybrid_io=0 llite.lustre-ffff9e5389b35000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.27529 s, 2.0 MB/s PASS 108a (7s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 14:44:14 (1778352254) 192.168.204.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg435-client.virtnet /mnt/lustre (opts:) 192.168.204.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg435-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg435-client.virtnet: -o user_xattr,flock 192.168.204.135@tcp:/lustre /mnt/lustre Starting client: oleg435-client.virtnet: -o user_xattr,flock 192.168.204.135@tcp:/lustre /mnt/lustre2 192.168.204.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg435-client.virtnet /mnt/lustre (opts:) Stopping client oleg435-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg435-server' oleg435-server: oleg435-server.virtnet: executing load_modules_local oleg435-server: Loading modules from /home/green/git/lustre-release/lustre oleg435-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg435-server: detected 4 online CPUs by sysfs oleg435-server: MODOPTS_LIBCFS= oleg435-server: Force libcfs to create 2 CPU partitions Iteration 1 Starting client: oleg435-client.virtnet: -o user_xattr,flock 192.168.204.135@tcp:/lustre /mnt/lustre Starting client: oleg435-client.virtnet: -o user_xattr,flock 192.168.204.135@tcp:/lustre /mnt/lustre2 192.168.204.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg435-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg435-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg435-server' oleg435-server: oleg435-server.virtnet: executing load_modules_local oleg435-server: Loading modules from /home/green/git/lustre-release/lustre oleg435-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg435-server: detected 4 online CPUs by sysfs oleg435-server: MODOPTS_LIBCFS= oleg435-server: Force libcfs to create 2 CPU partitions Iteration 2 Starting client: oleg435-client.virtnet: -o user_xattr,flock 192.168.204.135@tcp:/lustre /mnt/lustre Starting client: oleg435-client.virtnet: -o user_xattr,flock 192.168.204.135@tcp:/lustre /mnt/lustre2 192.168.204.135@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg435-client.virtnet /mnt/lustre (opts:) Stopping client oleg435-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg435-server' oleg435-server: oleg435-server.virtnet: executing load_modules_local oleg435-server: Loading modules from /home/green/git/lustre-release/lustre oleg435-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg435-server: detected 4 online CPUs by sysfs oleg435-server: MODOPTS_LIBCFS= oleg435-server: Force libcfs to create 2 CPU partitions Starting client: oleg435-client.virtnet: -o user_xattr,flock 192.168.204.135@tcp:/lustre /mnt/lustre Starting client: oleg435-client.virtnet: -o user_xattr,flock 192.168.204.135@tcp:/lustre /mnt/lustre2 PASS 109 (43s) == sanityn test 110: do not grant another lock on resend ========================================================== 14:44:57 (1778352297) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-09 14:44:58.000000000 -0400 Modify: 2026-05-09 14:44:58.000000000 -0400 Change: 2026-05-09 14:44:58.000000000 -0400 Birth: 2026-05-09 14:44:57.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg435-client.virtnet oleg435-client.virtnet PASS 110 (169s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 14:47:46 (1778352466) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (1s) == sanityn test 112: update max-inherit in default LMV === 14:47:48 (1778352468) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (2s) == sanityn test 113: check servers of specified fs ======= 14:47:49 (1778352469) lustre-OST0000-osc-ffff9e5387863000 active. lustre-OST0001-osc-ffff9e5387863000 active. lustre-OST0000-osc-ffff9e5387863000 active. lustre-OST0001-osc-ffff9e5387863000 active. MGC192.168.204.135@tcp active. PASS 113 (3s) == sanityn test 114: implicit default LMV inherit ======== 14:47:52 (1778352472) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (2s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 14:47:54 (1778352474) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (1s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 14:47:55 (1778352475) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (2s) == sanityn test 121: trunc append race =================== 14:47:57 (1778352477) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00461919 s, 4.4 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0263671 s, 777 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (5s) == sanityn test 200: service remains healthy while able to process request ========================================================== 14:48:02 (1778352482) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0989 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1098 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1059 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0969 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1011 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1002 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1038 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1391 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1502 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1623 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1602 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1706 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1569 s, 0.1 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1628 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2584 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2576 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2324 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2538 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg435-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9e5386056000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e5386056000.ost_server_uuid in IDLE state after 0 sec PASS 200 (70s) cleanup: ====================================================== == sanityn test complete, duration 5409 sec ============== 14:49:12 (1778352552) === sanityn: start cleanup 14:49:13 (1778352553) === Stopping clients: oleg435-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg435-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 14:50:28 (1778352628) ===