-----============= acceptance-small: sanityn ============----- Mon Dec 22 02:28:25 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg324-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a 102 skipping tests SLOW=no: 33a === sanityn: start setup 02:28:32 (1766388512) === Starting client oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 Mount client oleg324-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 Started clients oleg324-client.virtnet: 192.168.203.124@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg324-client.virtnet: executing check_config_client /mnt/lustre oleg324-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg324-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8cd00388b800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8cd018895000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8cd00388b800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8cd018895000.idle_timeout=debug disable quota as required oleg324-server: oleg324-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 02:28:46 (1766388526) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194592 s, 53.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7552] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 02:28:48 (1766388528) PASS 0 (8s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 02:28:56 (1766388536) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (7s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 02:29:03 (1766388543) -rw-r--r-- 1 root root 0 Dec 22 02:29 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (7s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 02:29:10 (1766388550) -rw-r--r-- 1 root root 0 Dec 22 02:29 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (6s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 02:29:16 (1766388556) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (6s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 02:29:22 (1766388562) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (4s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 02:29:26 (1766388566) total 1 -rwxrwxrwx 1 root root 0 Dec 22 02:29 f2a -rwxrwxrwx 1 root root 0 Dec 22 02:29 f2b total 1 -rwxrwxrwx 1 root root 0 Dec 22 02:29 f2a -rwxrwxrwx 1 root root 0 Dec 22 02:29 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.00832697 s, 61.5 kB/s PASS 2e (4s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 02:29:30 (1766388570) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (1s) == sanityn test 2g: check blocks update on sync write ==== 02:29:31 (1766388571) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.177729 s, 11.8 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (4s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 02:29:35 (1766388575) PASS 3 (2s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 02:29:37 (1766388577) PASS 4 (4s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 02:29:41 (1766388581) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (3s) == sanityn test 6: remove of open file on other node ============================================================================ 02:29:44 (1766388584) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (2s) == sanityn test 7: remove of open directory on other node ======================================================================= 02:29:46 (1766388586) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (2s) == sanityn test 8: remove of open special file on other node ==================================================================== 02:29:48 (1766388588) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (3s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 02:29:51 (1766388591) PASS 9a (3s) == sanityn test 9b: append to striped sparse file ======== 02:29:54 (1766388594) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206153 s, 50.9 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0198666 s, 0.2 kB/s Data read (expecting 'foo'): foo PASS 9b (3s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 02:29:57 (1766388597) 1+0 records in 1+0 records out 1 byte copied, 0.00482693 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00562369 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00510293 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00641439 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00663575 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0116348 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00890886 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00777584 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00610023 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0120638 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0120899 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00845779 s, 0.1 kB/s PASS 10a (3s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 02:30:00 (1766388600) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0096921 s, 317 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0215148 s, 190 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000245374 s, 12.5 MB/s PASS 10b (3s) == sanityn test 11: execution of file opened for write should return error ============================================================== 02:30:03 (1766388603) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7552 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 300: /mnt/lustre2/d11/f: Text file busy PASS 11 (3s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 02:30:06 (1766388606) warning: '-runtime' deprecated, use '-t runtime' instead - link 1181 (time 1766388617.47 total 10.00 last 118.08) - link 2177 (time 1766388627.48 total 20.02 last 99.46) - link 3273 (time 1766388637.49 total 30.02 last 109.57) - link 4286 (time 1766388647.49 total 40.02 last 101.27) - link 5847 (time 1766388657.49 total 50.03 last 156.05) - link 7249 (time 1766388667.50 total 60.03 last 140.11) - link 8353 (time 1766388677.51 total 70.04 last 110.34) - link 9553 (time 1766388687.51 total 80.04 last 120.00) - link 10000 (time 1766388691.25 total 83.78 last 119.37) - link 11292 (time 1766388701.26 total 93.79 last 129.11) total: 12147 link in 100.00 seconds: 121.46 ops/second using seed 584810912 running for 100 seconds - stat 10000 (time 1766388616 ; total 8 ; last 8) - stat 20000 (time 1766388624 ; total 16 ; last 8) - stat 30000 (time 1766388633 ; total 25 ; last 9) - stat 40000 (time 1766388641 ; total 33 ; last 8) - stat 50000 (time 1766388649 ; total 41 ; last 8) - stat 60000 (time 1766388657 ; total 49 ; last 8) - stat 66164 (time 1766388668 ; total 60 ; last 11) - stat 70000 (time 1766388675 ; total 67 ; last 7) - stat 75460 (time 1766388686 ; total 78 ; last 11) - stat 80000 (time 1766388694 ; total 86 ; last 8) - stat 85560 (time 1766388705 ; total 97 ; last 11) total: 87647 stats in 100 seconds: 876.469971 stats/second - unlinked 0 (time 1766388658 ; total 0 ; last 0) - unlinked 10000 (time 1766388754 ; total 96 ; last 96) unlink(/mnt/lustre2/lockdir/lockfile12147) error: No such file or directory total: 12146 unlinks in 110 seconds: 110.418182 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21328) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21331) - No such process PASS 12 (165s) == sanityn test 13: test directory page revocation ======= 02:32:51 (1766388771) f13 f13 PASS 13 (3s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 02:32:54 (1766388774) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7552 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 335: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 22520 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (3s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 02:32:57 (1766388777) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 23098 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (3s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 02:33:00 (1766388780) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 350: 23677 Terminated $DIR1/$tdir/sleep 60 PASS 14b (2s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 02:33:02 (1766388782) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 362: 24258 Terminated $DIR1/$tdir/sleep 60 PASS 14c (3s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 02:33:05 (1766388785) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 373: 24840 Terminated $DIR1/$tdir/sleep 60 PASS 14d (4s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 02:33:09 (1766388789) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7524352 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7524352KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (31s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 02:33:41 (1766388821) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.121589 s, 69.0 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.113445 s, 73.9 MB/s Chance of close/open is 1 in 50 Seed set to 8822 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read skipping zero size read 000011[1] 1766388822.678779 write 0x8b2b7c thru 0x8b5055 (0x24da bytes) 000012[0] 1766388822.681990 mapwrite 0x3c3263 thru 0x3d067b (0xd419 bytes) 000013[0] 1766388822.747157 mapread 0x190c67 thru 0x19d2cd (0xc667 bytes) 000015[1] 1766388822.783281 mapwrite 0x0eb833 thru 0x0ed720 (0x1eee bytes) 000016[1] 1766388822.840087 read 0x09200c thru 0x0927f4 (0x7e9 bytes) 000017[1] 1766388822.855604 mapread 0x829c99 thru 0x8399f4 (0xfd5c bytes) 000019[1] 1766388822.892475 mapwrite 0x2780ea thru 0x27c990 (0x48a7 bytes) 000022[1] 1766388822.950629 write 0x444b60 thru 0x451ca5 (0xd146 bytes) 000023[0] 1766388822.959034 write 0x869708 thru 0x87459e (0xae97 bytes) 000025[1] 1766388822.993089 mapwrite 0x9d926c thru 0x9e0dc0 (0x7b55 bytes) 000026[0] 1766388823.050539 mapwrite 0x5ee494 thru 0x5fb3b8 (0xcf25 bytes) 000027[0] 1766388823.116417 read 0x53b17f thru 0x545b52 (0xa9d4 bytes) 000030[0] 1766388823.143212 mapwrite 0x4989c7 thru 0x499d22 (0x135c bytes) 000032[0] 1766388823.189988 mapwrite 0x0c98c0 thru 0x0d627b (0xc9bc bytes) 000033[0] 1766388823.245525 mapwrite 0x6d08ec thru 0x6d0a9a (0x1af bytes) 000034[1] 1766388823.275665 mapread 0x560d62 thru 0x566fef (0x628e bytes) truncating to largest ever: 0x9dfb2e 000035[1] 1766388823.302002 trunc from 0x9e0dc1 to 0x9dfb2d (0x1293 bytes) 000036[0] 1766388823.319678 trunc from 0x9dfb2e to 0x0ed1f9 (0x8f2934 bytes) 000038[1] 1766388823.338489 write 0x82e28f thru 0x833b8f (0x5901 bytes) 000042[0] 1766388823.376483 mapread 0x0c065a thru 0x0c40d7 (0x3a7e bytes) 000044[0] 1766388823.415360 mapread 0x77d1c5 thru 0x77f282 (0x20be bytes) 000046[0] 1766388823.431973 write 0x3ba838 thru 0x3bbe0d (0x15d6 bytes) 000048[1] 1766388823.454102 mapread 0x3e6798 thru 0x3efb76 (0x93df bytes) 000051[1] 1766388823.492308 write 0x8ea7b5 thru 0x8ebee6 (0x1732 bytes) 000053[1] 1766388823.512611 trunc from 0x8ebee7 to 0x36b472 (0x580a74 bytes) 000054[0] 1766388823.524460 read 0x23aaff thru 0x24a9dc (0xfede bytes) 000055[1] 1766388823.541240 mapwrite 0x1eedb8 thru 0x1feccf (0xff18 bytes) 000056[1] 1766388823.603222 mapwrite 0x826f55 thru 0x8290a2 (0x214e bytes) 000057[1] 1766388823.649385 trunc from 0x8290a3 to 0x2ea597 (0x53eb0b bytes) 000058[1] 1766388823.657253 read 0x033efb thru 0x03e983 (0xaa89 bytes) 000060[0] 1766388823.706335 mapread 0x0cd458 thru 0x0d104f (0x3bf8 bytes) 000061[1] 1766388823.730431 trunc from 0x2ea598 to 0x1282e3 (0x1c22b4 bytes) 000063[1] 1766388823.756504 read 0x0523e3 thru 0x05d6a8 (0xb2c6 bytes) 000064[1] 1766388823.785486 read 0x009325 thru 0x009edf (0xbbb bytes) 000065[0] 1766388823.786143 write 0x9caaee thru 0x9d5afe (0xb011 bytes) 000066[1] 1766388823.808064 read 0x66c5ae thru 0x6734b4 (0x6f07 bytes) 000067[1] 1766388823.845990 trunc from 0x9d5aff to 0x3336b5 (0x6a2449 bytes) 000068[1] 1766388823.884258 trunc from 0x3336b6 to 0x7ac002 (0x47894d bytes) 000070[0] 1766388823.919271 mapwrite 0x884a32 thru 0x888405 (0x39d4 bytes) 000072[0] 1766388823.996840 trunc from 0x888406 to 0x5c367c (0x2c4d89 bytes) 000073[0] 1766388824.014681 mapwrite 0x9a4d06 thru 0x9b3548 (0xe843 bytes) 000074[0] 1766388824.091085 trunc from 0x9b3549 to 0x1b5af3 (0x7fda55 bytes) 000079[1] 1766388824.142459 write 0x973d99 thru 0x97fce1 (0xbf49 bytes) 000082[0] 1766388824.181131 mapwrite 0x7ef6bc thru 0x7fe230 (0xeb75 bytes) 000083[0] 1766388824.419188 mapwrite 0x85d68a thru 0x860007 (0x297e bytes) 000085[1] 1766388824.470173 mapwrite 0x235cfe thru 0x23d52f (0x7832 bytes) 000086[1] 1766388824.521860 trunc from 0x97fce2 to 0x19f0d5 (0x7e0c0c bytes) 000087[0] 1766388824.538768 read 0x0e5c0e thru 0x0ed619 (0x7a0c bytes) 000092[1] 1766388824.576802 write 0x4faf07 thru 0x509185 (0xe27f bytes) 000093[0] 1766388824.579103 read 0x35710a thru 0x3571a6 (0x9d bytes) 000096[0] 1766388824.598891 mapread 0x41fb24 thru 0x421482 (0x195f bytes) 000097[0] 1766388824.620456 trunc from 0x509186 to 0x50bb1f (0x299a bytes) 000098[1] 1766388824.647084 mapread 0x48b928 thru 0x48d8e4 (0x1fbd bytes) 000101[0] 1766388824.685969 mapwrite 0x8e1c76 thru 0x8ecd29 (0xb0b4 bytes) 000103[0] 1766388824.747467 trunc from 0x8ecd2a to 0x4f91b2 (0x3f3b77 bytes) 000104[1] 1766388824.763397 trunc from 0x4f91b3 to 0x8efb81 (0x3f69cf bytes) 000107[1] 1766388824.779666 mapwrite 0x0f258b thru 0x0fd4cf (0xaf45 bytes) 000108[1] 1766388824.817027 write 0x08d2d3 thru 0x095dfc (0x8b2a bytes) 000109[0] 1766388824.822633 write 0x400f75 thru 0x403768 (0x27f4 bytes) 000110[0] 1766388824.839198 trunc from 0x8efb82 to 0x84c71a (0xa3467 bytes) 000111[0] 1766388824.860887 mapwrite 0x1e73e9 thru 0x1f1790 (0xa3a8 bytes) 000112[1] 1766388824.897059 mapread 0x1562c5 thru 0x1645ab (0xe2e7 bytes) 000113[0] 1766388824.918010 write 0x8e7921 thru 0x8ec6d0 (0x4db0 bytes) 000114[0] 1766388824.923786 mapread 0x0e63c7 thru 0x0eb354 (0x4f8e bytes) 000116[0] 1766388824.933297 write 0x791110 thru 0x7a08cb (0xf7bc bytes) 000120[1] 1766388824.948032 write 0x4bbb03 thru 0x4c38f6 (0x7df4 bytes) 000121[0] 1766388824.966234 mapwrite 0x58c89f thru 0x59360f (0x6d71 bytes) 000122[0] 1766388825.023873 trunc from 0x8ec6d1 to 0x357bed (0x594ae3 bytes) 000126[0] 1766388825.050950 mapread 0x16bd00 thru 0x17b760 (0xfa61 bytes) 000128[0] 1766388825.074232 mapread 0x272d76 thru 0x27c869 (0x9af4 bytes) 000129[1] 1766388825.084535 trunc from 0x357bee to 0x92b95d (0x5d3d70 bytes) 000130[0] 1766388825.112288 mapwrite 0x096adf thru 0x09968c (0x2bae bytes) 000132[1] 1766388825.160020 mapwrite 0x2a3f08 thru 0x2af574 (0xb66d bytes) 000133[1] 1766388825.203042 write 0x037ba6 thru 0x0465a0 (0xe9fb bytes) 000134[1] 1766388825.218984 mapread 0x2c4348 thru 0x2c6be5 (0x289e bytes) 000136[0] 1766388825.230425 mapwrite 0x462ded thru 0x46e619 (0xb82d bytes) 000137[0] 1766388825.282903 trunc from 0x92b95e to 0x88addb (0xa0b82 bytes) 000140[1] 1766388825.314641 read 0x640eb5 thru 0x64b55c (0xa6a8 bytes) 000144[0] 1766388825.339812 write 0x584e53 thru 0x585881 (0xa2f bytes) 000146[1] 1766388825.356288 mapread 0x525b81 thru 0x531df6 (0xc276 bytes) 000149[0] 1766388825.389297 mapwrite 0x8dcc12 thru 0x8e9843 (0xcc32 bytes) 000150[0] 1766388825.447676 mapwrite 0x9d0c4d thru 0x9df3a9 (0xe75d bytes) 000153[0] 1766388825.542488 mapwrite 0x782eac thru 0x7882e9 (0x543e bytes) 000154[0] 1766388825.582503 trunc from 0x9df3aa to 0x24cdce (0x7925db bytes) 000155[0] 1766388825.606524 write 0x516ee0 thru 0x524c2c (0xdd4d bytes) 000156[0] 1766388825.612686 mapwrite 0x13e23f thru 0x14a84f (0xc611 bytes) 000159[1] 1766388825.695140 write 0x26b7f6 thru 0x2786e7 (0xcef2 bytes) 000160[0] 1766388825.712506 write 0x83cad0 thru 0x847112 (0xa643 bytes) 000161[1] 1766388825.724778 mapread 0x5e916f thru 0x5f5dcd (0xcc5f bytes) 000163[0] 1766388825.838178 write 0x22963b thru 0x22ae7c (0x1842 bytes) 000165[0] 1766388825.852700 mapread 0x1b93f5 thru 0x1c58d4 (0xc4e0 bytes) 000166[0] 1766388825.862837 read 0x313889 thru 0x3196c9 (0x5e41 bytes) 000167[0] 1766388825.874419 mapwrite 0x2ebf4b thru 0x2f6577 (0xa62d bytes) 000168[1] 1766388825.912182 read 0x346238 thru 0x34f006 (0x8dcf bytes) 000169[0] 1766388825.922983 write 0x741d35 thru 0x74f595 (0xd861 bytes) 000170[0] 1766388825.937850 mapread 0x0afd40 thru 0x0b6a00 (0x6cc1 bytes) 000171[0] 1766388825.950621 read 0x24e212 thru 0x24fb58 (0x1947 bytes) 000175[1] 1766388825.972774 mapwrite 0x71c472 thru 0x72a933 (0xe4c2 bytes) 000176[0] 1766388826.027849 mapread 0x2c4b28 thru 0x2cb10f (0x65e8 bytes) 000177[1] 1766388826.041986 write 0x86d4c8 thru 0x878663 (0xb19c bytes) 000178[0] 1766388826.054970 mapwrite 0x265b0d thru 0x2759e8 (0xfedc bytes) 000179[0] 1766388826.105575 write 0x44282b thru 0x4467bf (0x3f95 bytes) 000180[1] 1766388826.125736 read 0x35178d thru 0x35f078 (0xd8ec bytes) 000183[1] 1766388826.146480 mapwrite 0x1bd0a7 thru 0x1c3c6e (0x6bc8 bytes) 000184[0] 1766388826.199768 trunc from 0x878664 to 0x0af5e6 (0x7c907d bytes) 000186[0] 1766388826.227245 trunc from 0x0af5e7 to 0x10cafd (0x5d517 bytes) 000187[0] 1766388826.243099 read 0x107363 thru 0x1086bc (0x135a bytes) 000189[0] 1766388826.270518 write 0x2f6991 thru 0x30651a (0xfb8a bytes) 000190[1] 1766388826.274619 mapwrite 0x15b595 thru 0x1656af (0xa11b bytes) 000194[1] 1766388826.369444 mapwrite 0x5aab90 thru 0x5b3bec (0x905d bytes) 000195[0] 1766388826.432846 write 0x780b53 thru 0x78cbac (0xc05a bytes) 000196[0] 1766388826.449205 write 0x388cda thru 0x38d7c6 (0x4aed bytes) 000198[0] 1766388826.497351 read 0x4c73d5 thru 0x4d5020 (0xdc4c bytes) 000200[0] 1766388826.515105 write 0x50e6a3 thru 0x5139d7 (0x5335 bytes) 000201[1] 1766388826.525614 mapread 0x4c4708 thru 0x4cd284 (0x8b7d bytes) 000205[0] 1766388826.571087 mapread 0x316446 thru 0x31e7ef (0x83aa bytes) 000207[1] 1766388826.581318 mapread 0x26e318 thru 0x272dce (0x4ab7 bytes) 000209[0] 1766388826.618829 mapwrite 0x5f5676 thru 0x603fe7 (0xe972 bytes) 000210[0] 1766388826.671939 trunc from 0x78cbad to 0x3fa0d5 (0x392ad7 bytes) 000213[0] 1766388826.708116 write 0x1cf1e2 thru 0x1d2679 (0x3498 bytes) 000215[1] 1766388826.714395 read 0x36cf19 thru 0x372307 (0x53ef bytes) 000216[1] 1766388826.765993 write 0x3eee67 thru 0x3f8ece (0xa068 bytes) 000217[1] 1766388826.795531 mapwrite 0x1a67d1 thru 0x1b206a (0xb89a bytes) 000218[0] 1766388826.883887 write 0x849609 thru 0x8557c1 (0xc1b9 bytes) 000219[1] 1766388826.901172 mapwrite 0x7aa080 thru 0x7b3d14 (0x9c95 bytes) 000220[0] 1766388827.060631 trunc from 0x8557c2 to 0x2b54f0 (0x5a02d1 bytes) 000221[1] 1766388827.087367 read 0x09b087 thru 0x0a1d99 (0x6d13 bytes) 000223[0] 1766388827.135496 mapwrite 0x1438da thru 0x14b542 (0x7c69 bytes) 000224[0] 1766388827.175257 read 0x1c7167 thru 0x1d5824 (0xe6be bytes) 000227[0] 1766388827.208914 mapread 0x11b119 thru 0x11c0c8 (0xfb0 bytes) 000230[1] 1766388827.210209 trunc from 0x2b54f1 to 0x4340b6 (0x17ebc6 bytes) 000231[1] 1766388827.231810 write 0x032a40 thru 0x037e81 (0x5442 bytes) 000232[1] 1766388827.250167 read 0x2ee5c7 thru 0x2f6662 (0x809c bytes) 000233[0] 1766388827.250744 mapread 0x202f13 thru 0x208b39 (0x5c27 bytes) 000234[1] 1766388827.299171 trunc from 0x4340b7 to 0x0a7707 (0x38c9af bytes) 000236[0] 1766388827.314901 trunc from 0x0a7708 to 0x928968 (0x881261 bytes) 000238[1] 1766388827.343222 read 0x103277 thru 0x1038c2 (0x64c bytes) 000241[1] 1766388827.365317 write 0x7d57d7 thru 0x7e53c7 (0xfbf1 bytes) 000244[0] 1766388827.379591 mapread 0x5b2eb7 thru 0x5baae6 (0x7c30 bytes) 000246[0] 1766388827.440082 write 0x80905d thru 0x81461b (0xb5bf bytes) 000247[1] 1766388827.470759 mapwrite 0x35287d thru 0x35cd01 (0xa485 bytes) 000249[0] 1766388827.521012 write 0x68a907 thru 0x699998 (0xf092 bytes) 000250[0] 1766388827.540949 mapwrite 0x9ff7fd thru 0x9fffff (0x803 bytes) 000251[0] 1766388827.587906 read 0x9beafa thru 0x9c9621 (0xab28 bytes) 000252[0] 1766388827.590803 mapwrite 0x07361f thru 0x07ed94 (0xb776 bytes) 000254[1] 1766388827.638378 mapread 0x1525a2 thru 0x1587f5 (0x6254 bytes) 000255[1] 1766388827.663706 mapread 0x0c4a17 thru 0x0c6cc4 (0x22ae bytes) 000256[1] 1766388827.677643 trunc from 0xa00000 to 0x50709e (0x4f8f61 bytes) 000257[1] 1766388827.705079 read 0x1f7b4f thru 0x1fa732 (0x2be4 bytes) 000259[1] 1766388827.718744 mapread 0x4708ae thru 0x47ec9d (0xe3f0 bytes) 000261[0] 1766388827.742303 mapread 0x15202d thru 0x15b7d3 (0x97a7 bytes) 000263[0] 1766388827.771243 write 0x5b0df3 thru 0x5bc7e8 (0xb9f6 bytes) 000264[1] 1766388827.785463 trunc from 0x5bc7e9 to 0x1b7492 (0x405356 bytes) 000268[1] 1766388827.825500 mapread 0x19128f thru 0x198c75 (0x79e7 bytes) 000270[0] 1766388827.841821 read 0x181f03 thru 0x186183 (0x4281 bytes) 000271[1] 1766388827.867927 mapwrite 0x9342fb thru 0x93da13 (0x9719 bytes) 000272[1] 1766388827.923773 write 0x1f766e thru 0x204d85 (0xd718 bytes) 000273[0] 1766388827.938087 mapwrite 0x823be0 thru 0x83313e (0xf55f bytes) 000274[0] 1766388828.048588 mapwrite 0x040421 thru 0x0406a8 (0x288 bytes) 000275[0] 1766388828.078948 mapread 0x93c9ca thru 0x93da13 (0x104a bytes) 000276[1] 1766388828.098613 mapwrite 0x5e5a1e thru 0x5ead55 (0x5338 bytes) 000277[1] 1766388828.137503 write 0x235ccd thru 0x23dbe9 (0x7f1d bytes) 000278[1] 1766388828.154904 write 0x397ed8 thru 0x3a6616 (0xe73f bytes) 000281[0] 1766388828.168756 mapread 0x7257c6 thru 0x732d86 (0xd5c1 bytes) 000283[1] 1766388828.218890 trunc from 0x93da14 to 0x0732a1 (0x8ca772 bytes) 000284[0] 1766388828.241427 write 0x4336c2 thru 0x44133a (0xdc79 bytes) 000285[0] 1766388828.254481 read 0x02cab8 thru 0x0338e3 (0x6e2c bytes) 000290[0] 1766388828.283683 trunc from 0x44133b to 0x3fcb7a (0x447c0 bytes) 000291[0] 1766388828.306162 trunc from 0x3fcb7b to 0x78fd7f (0x393205 bytes) 000292[0] 1766388828.325054 mapread 0x5478c0 thru 0x54a23b (0x297c bytes) 000293[0] 1766388828.352619 mapread 0x508e45 thru 0x50c2c6 (0x3482 bytes) 000294[1] 1766388828.355654 mapread 0x40eb6e thru 0x41dcce (0xf161 bytes) 000295[0] 1766388828.381846 mapread 0x50aaab thru 0x51302d (0x8583 bytes) 000298[0] 1766388828.415074 mapwrite 0x49f73e thru 0x4a8edd (0x97a0 bytes) 000299[1] 1766388828.461471 mapread 0x0a0e16 thru 0x0a40d1 (0x32bc bytes) 000302[0] 1766388828.506108 trunc from 0x78fd80 to 0x03ce7f (0x752f00 bytes) 000303[1] 1766388828.529696 read 0x030ba6 thru 0x03752d (0x6988 bytes) 000310[0] 1766388828.566239 write 0x2920e7 thru 0x29fd9e (0xdcb8 bytes) 000311[0] 1766388828.572522 mapwrite 0x1685e1 thru 0x171727 (0x9147 bytes) 000314[0] 1766388828.619840 mapread 0x276ed2 thru 0x284a21 (0xdb50 bytes) 000317[1] 1766388828.651504 trunc from 0x29fd9f to 0x718ee4 (0x479146 bytes) 000320[1] 1766388828.692041 mapread 0x293a0f thru 0x2a1d36 (0xe328 bytes) 000323[0] 1766388828.744994 read 0x4207e0 thru 0x425889 (0x50aa bytes) 000324[0] 1766388828.755832 read 0x611a24 thru 0x616524 (0x4b01 bytes) 000325[1] 1766388828.760133 mapread 0x04ae9d thru 0x05ac49 (0xfdad bytes) 000327[0] 1766388828.768028 read 0x2c6f35 thru 0x2cbee3 (0x4faf bytes) 000328[1] 1766388828.780078 mapread 0x367505 thru 0x36d150 (0x5c4c bytes) 000330[1] 1766388828.799537 write 0x0dc14e thru 0x0dc860 (0x713 bytes) 000332[1] 1766388828.816266 mapwrite 0x3f780c thru 0x3fc4c8 (0x4cbd bytes) 000333[1] 1766388828.862261 mapwrite 0x64c6ca thru 0x6571fe (0xab35 bytes) 000335[0] 1766388828.936274 write 0x679205 thru 0x681ff5 (0x8df1 bytes) 000339[1] 1766388828.985074 mapwrite 0x691fea thru 0x694516 (0x252d bytes) 000341[0] 1766388829.046225 read 0x30ce9e thru 0x30eec8 (0x202b bytes) 000343[0] 1766388829.072794 write 0x51e4e1 thru 0x525212 (0x6d32 bytes) 000344[1] 1766388829.092768 mapread 0x13770c thru 0x13f916 (0x820b bytes) 000345[0] 1766388829.128328 mapwrite 0x171a89 thru 0x17b540 (0x9ab8 bytes) 000347[0] 1766388829.168191 mapwrite 0x70b346 thru 0x716e4f (0xbb0a bytes) 000349[0] 1766388829.202208 trunc from 0x718ee5 to 0x89af40 (0x18205c bytes) 000352[0] 1766388829.225139 write 0x81484d thru 0x82331b (0xeacf bytes) 000356[1] 1766388829.252932 mapwrite 0x189fce thru 0x18c75f (0x2792 bytes) 000359[0] 1766388829.313320 mapread 0x59d3fd thru 0x5ac204 (0xee08 bytes) 000360[0] 1766388829.337210 read 0x2118cb thru 0x21b1e4 (0x991a bytes) 000361[1] 1766388829.345445 mapread 0x76dd0e thru 0x778cfa (0xafed bytes) 000368[0] 1766388829.368252 mapwrite 0x679f01 thru 0x689bbb (0xfcbb bytes) 000370[1] 1766388829.409968 read 0x1c8e8d thru 0x1d8acc (0xfc40 bytes) 000372[0] 1766388829.415413 trunc from 0x89af41 to 0x337f59 (0x562fe7 bytes) 000378[0] 1766388829.447890 read 0x002d28 thru 0x01150a (0xe7e3 bytes) 000379[0] 1766388829.481958 write 0x6655d4 thru 0x66a40e (0x4e3b bytes) 000380[0] 1766388829.492352 write 0x5a6886 thru 0x5ae066 (0x77e1 bytes) 000381[0] 1766388829.498506 mapread 0x4d6de9 thru 0x4e5ec2 (0xf0da bytes) 000382[0] 1766388829.555612 trunc from 0x66a40f to 0x8bc729 (0x25231b bytes) 000386[0] 1766388829.595401 read 0x6c8135 thru 0x6ca987 (0x2853 bytes) 000387[0] 1766388829.598743 mapread 0x4b6f29 thru 0x4c1a4c (0xab24 bytes) 000391[0] 1766388829.638947 trunc from 0x8bc72a to 0x91c71d (0x5fff4 bytes) 000392[0] 1766388829.656246 write 0x298b85 thru 0x29e047 (0x54c3 bytes) 000393[0] 1766388829.664034 mapwrite 0x4116f7 thru 0x41bce6 (0xa5f0 bytes) 000396[0] 1766388829.706408 mapwrite 0x9fb532 thru 0x9fbf42 (0xa11 bytes) 000397[0] 1766388829.743194 write 0x3e7d82 thru 0x3eeedc (0x715b bytes) 000398[0] 1766388829.751518 write 0x1b1884 thru 0x1b5bbe (0x433b bytes) 000400[1] 1766388829.767361 mapwrite 0x2b0c19 thru 0x2b9306 (0x86ee bytes) 000403[0] 1766388829.838485 mapread 0x16a523 thru 0x17a2fa (0xfdd8 bytes) 000406[0] 1766388829.868775 write 0x26f953 thru 0x274b81 (0x522f bytes) 000407[0] 1766388829.887170 write 0x06bf4e thru 0x071a84 (0x5b37 bytes) 000409[1] 1766388829.909270 trunc from 0x9fbf43 to 0x9d72f9 (0x24c49 bytes) 000410[1] 1766388829.933086 mapwrite 0x37028d thru 0x37ecff (0xea73 bytes) 000412[0] 1766388829.987793 read 0x7592bf thru 0x765768 (0xc4aa bytes) 000413[1] 1766388830.004233 mapread 0x5d24f1 thru 0x5dd791 (0xb2a1 bytes) 000415[0] 1766388830.038672 mapread 0x8e2c44 thru 0x8f245f (0xf81c bytes) 000416[0] 1766388830.061835 mapwrite 0x2afec9 thru 0x2b8689 (0x87c1 bytes) 000417[0] 1766388830.107081 mapwrite 0x84b5be thru 0x84f4ef (0x3f32 bytes) 000419[0] 1766388830.148082 trunc from 0x9d72fa to 0x9599c1 (0x7d938 bytes) 000421[0] 1766388830.173761 read 0x94d84f thru 0x94dd86 (0x538 bytes) 000422[0] 1766388830.184859 mapwrite 0x6e1264 thru 0x6e3f4e (0x2ceb bytes) 000423[0] 1766388830.219907 mapread 0x37642d thru 0x377b8a (0x175e bytes) 000424[1] 1766388830.236840 mapwrite 0x08de0f thru 0x08dff0 (0x1e2 bytes) 000427[1] 1766388830.300047 read 0x335112 thru 0x33a429 (0x5318 bytes) 000431[0] 1766388830.345406 trunc from 0x9599c2 to 0x337785 (0x62223c bytes) 000433[1] 1766388830.373048 write 0x249a07 thru 0x254455 (0xaa4f bytes) 000437[0] 1766388830.413603 mapwrite 0x7d16e9 thru 0x7d4bbc (0x34d4 bytes) 000439[1] 1766388830.478937 write 0x37f940 thru 0x38c878 (0xcf39 bytes) 000444[1] 1766388830.534679 trunc from 0x7d4bbd to 0x82c4e9 (0x5792d bytes) 000447[1] 1766388830.561824 write 0x06e800 thru 0x07864f (0x9e50 bytes) 000449[0] 1766388830.575121 read 0x5f0525 thru 0x5fa99a (0xa476 bytes) 000450[0] 1766388830.591614 write 0x3688d1 thru 0x36cafa (0x422a bytes) 000451[1] 1766388830.611526 mapread 0x615f1c thru 0x61a0f5 (0x41da bytes) 000452[1] 1766388830.635732 mapwrite 0x25d36b thru 0x25f32d (0x1fc3 bytes) 000453[0] 1766388830.683845 read 0x20830b thru 0x20efaa (0x6ca0 bytes) 000454[1] 1766388830.695144 write 0x6e9554 thru 0x6f1a05 (0x84b2 bytes) 000458[1] 1766388830.716600 write 0x3470b1 thru 0x34cdc4 (0x5d14 bytes) 000459[1] 1766388830.736993 mapwrite 0x74b443 thru 0x74be5d (0xa1b bytes) 000461[1] 1766388830.778410 mapread 0x46d1e3 thru 0x476ac7 (0x98e5 bytes) 000469[0] 1766388830.857224 trunc from 0x82c4ea to 0x46829c (0x3c424d bytes) 000471[1] 1766388830.898384 mapwrite 0x77e2a6 thru 0x788e0e (0xab69 bytes) 000474[1] 1766388830.968397 mapread 0x6e9043 thru 0x6f50b4 (0xc072 bytes) 000477[1] 1766388830.985523 mapwrite 0x5e7360 thru 0x5f0ca2 (0x9943 bytes) 000480[1] 1766388831.024896 write 0x4e9423 thru 0x4e94f1 (0xcf bytes) 000481[1] 1766388831.028624 write 0x87f4ae thru 0x88dee2 (0xea35 bytes) 000482[0] 1766388831.030246 trunc from 0x88dee3 to 0x8771df (0x16d03 bytes) 000484[0] 1766388831.073845 write 0x83a36c thru 0x83f46d (0x5102 bytes) 000486[1] 1766388831.081743 mapread 0x033752 thru 0x03fea9 (0xc758 bytes) 000491[0] 1766388831.159502 read 0x8635b4 thru 0x86eb0a (0xb557 bytes) 000493[1] 1766388831.180458 mapread 0x065191 thru 0x0669ba (0x182a bytes) 000497[1] 1766388831.191231 trunc from 0x8771e0 to 0x7423db (0x134e04 bytes) 000499[1] 1766388831.219792 write 0x104e8b thru 0x105046 (0x1bc bytes) 000500[1] 1766388831.232256 read 0x21e3f0 thru 0x2217d1 (0x33e2 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 8831 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x26222d 000016[1] 1766388831.651031 trunc from 00000000 to 0x26222c (0x26222d bytes) 000018[0] 1766388831.671708 mapwrite 0x9d8000 thru 0x9e1f99 (0x9f9a bytes) 000020[0] 1766388831.722265 mapwrite 0x428000 thru 0x4314e0 (0x94e1 bytes) 000021[1] 1766388831.760819 mapread 0x766000 thru 0x7686ce (0x26cf bytes) 000023[0] 1766388831.784912 write_OD 0x435000 thru 0x43efff (0xa000 bytes) 000027[0] 1766388831.814975 write_OD 0x2bb000 thru 0x2bdfff (0x3000 bytes) 000029[0] 1766388831.836168 read_OD 0x716000 thru 0x71ffff (0xa000 bytes) 000030[1] 1766388831.842129 write_OD 0x147000 thru 0x147fff (0x1000 bytes) truncating to largest ever: 0x4caf41 000031[0] 1766388831.864212 trunc from 0x9e1f9a to 0x4caf40 (0x517059 bytes) truncating to largest ever: 0x856c22 000033[0] 1766388831.887115 trunc from 0x4caf41 to 0x856c21 (0x38bce1 bytes) 000034[1] 1766388831.907220 mapread 0x733000 thru 0x73f607 (0xc608 bytes) 000038[1] 1766388831.930088 trunc from 0x856c22 to 0x6b59ce (0x1a1253 bytes) 000039[1] 1766388831.952313 read_OD 0x0a4000 thru 0x0abfff (0x8000 bytes) 000040[0] 1766388831.961159 mapread 0x0c5000 thru 0x0cbe24 (0x6e25 bytes) 000042[0] 1766388831.978246 write_OD 0x8fc000 thru 0x905fff (0xa000 bytes) 000044[1] 1766388832.005347 write_OD 0x444000 thru 0x44efff (0xb000 bytes) 000046[0] 1766388832.032413 mapread 0x0ea000 thru 0x0f1eb2 (0x7eb3 bytes) 000048[0] 1766388832.038068 mapwrite 0x836000 thru 0x842035 (0xc036 bytes) 000049[1] 1766388832.077863 trunc from 0x906000 to 0x79adcf (0x16b230 bytes) 000051[1] 1766388832.103334 write_OD 0x68f000 thru 0x68ffff (0x1000 bytes) 000052[0] 1766388832.131211 mapread 0x0ee000 thru 0x0f906d (0xb06e bytes) 000054[0] 1766388832.148896 write_OD 0x288000 thru 0x293fff (0xc000 bytes) 000058[1] 1766388832.177881 write_OD 0x613000 thru 0x61efff (0xc000 bytes) 000059[0] 1766388832.203454 mapread 0x713000 thru 0x71d099 (0xa09a bytes) 000061[0] 1766388832.214297 trunc from 0x79add0 to 0x6b2121 (0xe8cae bytes) 000064[0] 1766388832.243086 mapwrite 0x284000 thru 0x28c8a4 (0x88a5 bytes) 000066[0] 1766388832.290104 mapread 0x58e000 thru 0x59959b (0xb59c bytes) 000067[0] 1766388832.313060 write_OD 0x1d4000 thru 0x1ddfff (0xa000 bytes) 000072[1] 1766388832.335425 read_OD 0x16e000 thru 0x17afff (0xd000 bytes) 000074[0] 1766388832.340010 write_OD 0x458000 thru 0x45ffff (0x8000 bytes) 000076[1] 1766388832.360292 mapread 0x385000 thru 0x38cf2c (0x7f2d bytes) 000078[0] 1766388832.371951 write_OD 0x875000 thru 0x878fff (0x4000 bytes) 000079[0] 1766388832.389629 mapwrite 0x170000 thru 0x17f502 (0xf503 bytes) 000082[0] 1766388832.430464 mapread 0x5be000 thru 0x5c2833 (0x4834 bytes) 000083[0] 1766388832.438486 mapread 0x4b3000 thru 0x4b92dd (0x62de bytes) 000084[0] 1766388832.451470 write_OD 0x218000 thru 0x218fff (0x1000 bytes) 000085[0] 1766388832.469024 mapwrite 0x99a000 thru 0x9a0a3e (0x6a3f bytes) 000086[1] 1766388832.504864 write_OD 0x424000 thru 0x432fff (0xf000 bytes) 000087[0] 1766388832.522788 read_OD 0x05e000 thru 0x067fff (0xa000 bytes) 000088[1] 1766388832.526835 read_OD 0x825000 thru 0x832fff (0xe000 bytes) 000089[1] 1766388832.534116 mapread 0x831000 thru 0x836efa (0x5efb bytes) 000092[0] 1766388832.542196 mapread 0x361000 thru 0x3649a3 (0x39a4 bytes) 000098[1] 1766388832.551008 mapwrite 0x927000 thru 0x931b95 (0xab96 bytes) 000099[0] 1766388832.588726 mapwrite 0x54d000 thru 0x558008 (0xb009 bytes) 000100[0] 1766388832.626029 read_OD 0x22f000 thru 0x236fff (0x8000 bytes) 000101[0] 1766388832.633593 trunc from 0x9a0a3f to 0x49867d (0x5083c1 bytes) 000102[0] 1766388832.645085 mapwrite 0x2c6000 thru 0x2caec0 (0x4ec1 bytes) 000104[0] 1766388832.674691 write_OD 0x83c000 thru 0x843fff (0x8000 bytes) 000105[1] 1766388832.698717 read_OD 0x0f4000 thru 0x0fdfff (0xa000 bytes) 000106[0] 1766388832.702936 read_OD 0x70c000 thru 0x70efff (0x3000 bytes) 000107[0] 1766388832.707422 mapread 0x27f000 thru 0x288c20 (0x9c21 bytes) 000108[1] 1766388832.716380 mapwrite 0x764000 thru 0x765a6e (0x1a6f bytes) 000109[1] 1766388832.748459 write_OD 0x9a3000 thru 0x9acfff (0xa000 bytes) 000111[1] 1766388832.770114 trunc from 0x9ad000 to 0x780ee4 (0x22c11b bytes) 000114[0] 1766388832.795176 write_OD 0x3af000 thru 0x3bcfff (0xe000 bytes) truncating to largest ever: 0x9d78b1 000115[1] 1766388832.818017 trunc from 0x780ee5 to 0x9d78b0 (0x2569cc bytes) 000117[1] 1766388832.843701 mapwrite 0x23b000 thru 0x243c55 (0x8c56 bytes) 000120[0] 1766388832.890757 read_OD 0x14d000 thru 0x154fff (0x8000 bytes) 000124[1] 1766388832.904781 mapwrite 0x908000 thru 0x917d47 (0xfd48 bytes) 000125[0] 1766388832.946250 trunc from 0x9d78b1 to 0x40effb (0x5c88b5 bytes) 000126[1] 1766388832.958800 read_OD 0x326000 thru 0x327fff (0x2000 bytes) 000127[0] 1766388832.974429 trunc from 0x40effc to 0x2fa4aa (0x114b51 bytes) 000129[1] 1766388832.988521 mapread 0x1d8000 thru 0x1d85cd (0x5ce bytes) 000130[1] 1766388833.001094 mapread 0x16d000 thru 0x17cf4a (0xff4b bytes) 000134[1] 1766388833.017341 read_OD 0x0c5000 thru 0x0d3fff (0xf000 bytes) 000137[0] 1766388833.022872 read_OD 0x2d8000 thru 0x2e3fff (0xc000 bytes) 000142[0] 1766388833.028410 read_OD 0x0e6000 thru 0x0eafff (0x5000 bytes) 000143[1] 1766388833.032821 write_OD 0x387000 thru 0x391fff (0xb000 bytes) 000147[1] 1766388833.062337 write_OD 0x897000 thru 0x8a0fff (0xa000 bytes) 000149[1] 1766388833.085796 mapread 0x318000 thru 0x31b1b7 (0x31b8 bytes) 000151[0] 1766388833.125641 trunc from 0x8a1000 to 0x6ada45 (0x1f35ba bytes) 000152[1] 1766388833.142774 trunc from 0x6ada46 to 0x6a97e4 (0x4261 bytes) 000153[1] 1766388833.160967 mapwrite 0x844000 thru 0x8511c6 (0xd1c7 bytes) 000154[1] 1766388833.224253 trunc from 0x8511c7 to 0x5d3136 (0x27e090 bytes) 000155[0] 1766388833.240594 write_OD 0x84c000 thru 0x84dfff (0x2000 bytes) 000157[0] 1766388833.266806 write_OD 0x01a000 thru 0x01cfff (0x3000 bytes) 000159[1] 1766388833.287175 write_OD 0x2ab000 thru 0x2b7fff (0xd000 bytes) 000161[1] 1766388833.307680 trunc from 0x84e000 to 0x1301cc (0x71de33 bytes) 000162[0] 1766388833.323325 mapread 0x01f000 thru 0x02146f (0x2470 bytes) 000167[1] 1766388833.349731 trunc from 0x1301cd to 0x6cab18 (0x59a94c bytes) 000168[0] 1766388833.371325 mapwrite 0x610000 thru 0x61ac5f (0xac60 bytes) 000169[0] 1766388833.448436 mapread 0x682000 thru 0x68fed1 (0xded2 bytes) 000171[1] 1766388833.449301 read_OD 0x0a9000 thru 0x0b5fff (0xd000 bytes) 000173[1] 1766388833.468772 trunc from 0x6cab19 to 0x279882 (0x451296 bytes) 000175[0] 1766388833.492263 mapwrite 0x757000 thru 0x76615a (0xf15b bytes) 000179[1] 1766388833.549615 mapread 0x41d000 thru 0x427d32 (0xad33 bytes) 000180[0] 1766388833.569510 mapwrite 0x2c1000 thru 0x2c5642 (0x4643 bytes) 000182[0] 1766388833.620073 mapwrite 0x5cb000 thru 0x5ceda5 (0x3da6 bytes) 000185[0] 1766388833.664248 trunc from 0x76615b to 0x620cf3 (0x145467 bytes) 000186[0] 1766388833.673294 write_OD 0x4cd000 thru 0x4d0fff (0x4000 bytes) 000189[1] 1766388833.696642 read_OD 0x17f000 thru 0x189fff (0xb000 bytes) 000191[1] 1766388833.701110 write_OD 0x67e000 thru 0x688fff (0xb000 bytes) 000193[0] 1766388833.722649 mapwrite 0x34f000 thru 0x358b10 (0x9b11 bytes) 000194[0] 1766388833.776380 mapread 0x5b5000 thru 0x5b65c5 (0x15c6 bytes) 000196[0] 1766388833.781937 trunc from 0x689000 to 0x4e0273 (0x1a8d8c bytes) 000200[1] 1766388833.791152 mapwrite 0x92e000 thru 0x92f8c1 (0x18c2 bytes) 000204[0] 1766388833.840978 write_OD 0x1f3000 thru 0x201fff (0xf000 bytes) 000206[1] 1766388833.865039 mapwrite 0x36d000 thru 0x376b38 (0x9b39 bytes) 000207[1] 1766388833.898258 mapwrite 0x1fa000 thru 0x208812 (0xe813 bytes) 000208[1] 1766388833.946138 trunc from 0x92f8c2 to 0x7c79e4 (0x167edd bytes) 000210[1] 1766388833.959267 read_OD 0x0a4000 thru 0x0abfff (0x8000 bytes) 000212[0] 1766388833.970026 read_OD 0x172000 thru 0x17ffff (0xe000 bytes) 000216[0] 1766388833.979877 mapwrite 0x7d9000 thru 0x7d9ee0 (0xee1 bytes) 000217[1] 1766388834.022876 write_OD 0x1be000 thru 0x1bffff (0x2000 bytes) 000218[1] 1766388834.048290 mapwrite 0x240000 thru 0x24beeb (0xbeec bytes) 000220[1] 1766388834.097428 write_OD 0x437000 thru 0x444fff (0xe000 bytes) 000221[0] 1766388834.123876 trunc from 0x7d9ee1 to 0x500bd3 (0x2d930d bytes) 000223[1] 1766388834.147084 read_OD 0x29a000 thru 0x29cfff (0x3000 bytes) 000225[1] 1766388834.160998 mapwrite 0x5b7000 thru 0x5bd61f (0x6620 bytes) 000226[0] 1766388834.213050 mapwrite 0x3df000 thru 0x3e959a (0xa59b bytes) 000227[1] 1766388834.270958 trunc from 0x5bd620 to 0x3d823c (0x1e53e3 bytes) 000229[1] 1766388834.308953 mapwrite 0x812000 thru 0x815299 (0x329a bytes) 000230[1] 1766388834.357686 mapread 0x2ea000 thru 0x2f1d95 (0x7d96 bytes) 000232[0] 1766388834.380725 mapwrite 0x2b3000 thru 0x2b8954 (0x5955 bytes) 000235[1] 1766388834.437311 write_OD 0x157000 thru 0x163fff (0xd000 bytes) 000237[0] 1766388834.462717 mapwrite 0x07d000 thru 0x080d9e (0x3d9f bytes) 000240[0] 1766388834.515663 mapread 0x6d6000 thru 0x6e2f54 (0xcf55 bytes) 000241[0] 1766388834.541018 trunc from 0x81529a to 0x6b61b6 (0x15f0e3 bytes) 000242[0] 1766388834.559512 mapread 0x0dc000 thru 0x0e2c02 (0x6c03 bytes) 000243[0] 1766388834.573845 mapwrite 0x670000 thru 0x67c01b (0xc01c bytes) 000250[1] 1766388834.637585 trunc from 0x6b61b7 to 0x54314f (0x173067 bytes) 000251[1] 1766388834.654641 read_OD 0x172000 thru 0x173fff (0x2000 bytes) 000253[0] 1766388834.666265 mapread 0x050000 thru 0x05506a (0x506b bytes) 000254[1] 1766388834.681641 trunc from 0x543150 to 0x1e75d9 (0x35bb76 bytes) 000255[0] 1766388834.710744 read_OD 0x0ff000 thru 0x104fff (0x6000 bytes) 000257[1] 1766388834.725411 mapwrite 0x2a7000 thru 0x2b6d6f (0xfd70 bytes) 000258[1] 1766388834.781758 mapwrite 0x82a000 thru 0x82dddc (0x3ddd bytes) 000263[1] 1766388834.847672 read_OD 0x3db000 thru 0x3e0fff (0x6000 bytes) 000264[0] 1766388834.856985 trunc from 0x82dddd to 0x382cae (0x4ab12e bytes) 000266[0] 1766388834.870022 mapwrite 0x254000 thru 0x25e2ad (0xa2ae bytes) 000267[1] 1766388834.931856 write_OD 0x011000 thru 0x01bfff (0xb000 bytes) 000268[0] 1766388834.961842 read_OD 0x290000 thru 0x29bfff (0xc000 bytes) 000269[0] 1766388834.967173 trunc from 0x382caf to 0x73348c (0x3b07de bytes) 000273[1] 1766388835.001667 mapread 0x3d5000 thru 0x3dd6ec (0x86ed bytes) 000274[0] 1766388835.023206 mapwrite 0x176000 thru 0x17a637 (0x4638 bytes) 000278[0] 1766388835.074115 write_OD 0x3da000 thru 0x3dafff (0x1000 bytes) 000280[1] 1766388835.100223 trunc from 0x73348d to 0x70affc (0x28490 bytes) 000281[1] 1766388835.113128 mapwrite 0x86b000 thru 0x875990 (0xa991 bytes) 000286[1] 1766388835.162109 read_OD 0x234000 thru 0x234fff (0x1000 bytes) 000287[0] 1766388835.170233 mapread 0x14c000 thru 0x156c0e (0xac0f bytes) 000288[1] 1766388835.194191 mapwrite 0x202000 thru 0x2030d8 (0x10d9 bytes) 000289[1] 1766388835.220220 mapread 0x58a000 thru 0x5975cf (0xd5d0 bytes) 000290[1] 1766388835.241055 read_OD 0x569000 thru 0x56afff (0x2000 bytes) 000291[0] 1766388835.248746 mapwrite 0x5ec000 thru 0x5fa4a9 (0xe4aa bytes) 000292[1] 1766388835.317796 mapwrite 0x8c8000 thru 0x8c9ed8 (0x1ed9 bytes) 000293[1] 1766388835.360206 mapread 0x6ff000 thru 0x7056a0 (0x66a1 bytes) 000295[0] 1766388835.383690 mapwrite 0x159000 thru 0x15964a (0x64b bytes) 000297[0] 1766388835.435133 trunc from 0x8c9ed9 to 0x79c044 (0x12de94 bytes) 000300[0] 1766388835.445911 trunc from 0x79c045 to 0x381e2f (0x41a215 bytes) 000302[0] 1766388835.454868 trunc from 0x381e30 to 0x28ec99 (0xf3196 bytes) 000305[1] 1766388835.480213 write_OD 0x26e000 thru 0x273fff (0x6000 bytes) 000308[0] 1766388835.507157 mapwrite 0x5b1000 thru 0x5baca8 (0x9ca9 bytes) 000310[1] 1766388835.568204 trunc from 0x5baca9 to 0x33a807 (0x2804a1 bytes) 000311[1] 1766388835.587778 read_OD 0x10b000 thru 0x115fff (0xb000 bytes) 000312[1] 1766388835.597134 trunc from 0x33a808 to 0x9c61d0 (0x68b9c9 bytes) 000313[1] 1766388835.611401 write_OD 0x286000 thru 0x28afff (0x5000 bytes) 000316[0] 1766388835.636203 read_OD 0x88f000 thru 0x890fff (0x2000 bytes) 000320[0] 1766388835.640036 trunc from 0x9c61d1 to 0x8981e1 (0x12dfef bytes) 000321[0] 1766388835.653472 mapwrite 0x646000 thru 0x64cc68 (0x6c69 bytes) 000323[1] 1766388835.685894 trunc from 0x8981e2 to 0x117a2c (0x7807b5 bytes) 000324[0] 1766388835.700834 mapwrite 0x116000 thru 0x125730 (0xf731 bytes) 000325[1] 1766388835.776049 mapwrite 0x396000 thru 0x39a5d1 (0x45d2 bytes) 000327[1] 1766388835.856416 trunc from 0x39a5d2 to 0x88f534 (0x4f4f63 bytes) 000328[0] 1766388835.877920 mapread 0x121000 thru 0x1217d1 (0x7d2 bytes) 000331[1] 1766388835.903662 mapwrite 0x4e4000 thru 0x4ebe22 (0x7e23 bytes) 000333[0] 1766388835.979047 write_OD 0x9bb000 thru 0x9c0fff (0x6000 bytes) 000335[1] 1766388836.011102 read_OD 0x84d000 thru 0x853fff (0x7000 bytes) 000336[1] 1766388836.017707 trunc from 0x9c1000 to 0x449399 (0x577c66 bytes) 000337[1] 1766388836.041090 mapwrite 0x45f000 thru 0x464774 (0x5775 bytes) 000340[0] 1766388836.105204 read_OD 0x357000 thru 0x365fff (0xf000 bytes) 000342[1] 1766388836.123037 trunc from 0x464775 to 0x02fb3f (0x434c35 bytes) 000343[0] 1766388836.141720 mapread 0x024000 thru 0x02eabf (0xaac0 bytes) 000350[0] 1766388836.172181 read_OD 0x025000 thru 0x025fff (0x1000 bytes) 000351[0] 1766388836.175451 mapread 0x01e000 thru 0x02d35a (0xf35b bytes) 000352[1] 1766388836.181200 read_OD 0x029000 thru 0x02efff (0x6000 bytes) 000353[0] 1766388836.186184 read_OD 0x001000 thru 0x002fff (0x2000 bytes) 000359[1] 1766388836.191110 write_OD 0x30f000 thru 0x310fff (0x2000 bytes) 000360[0] 1766388836.223764 trunc from 0x311000 to 0x8dc027 (0x5cb028 bytes) 000363[0] 1766388836.257454 mapwrite 0x0e5000 thru 0x0f32c4 (0xe2c5 bytes) 000364[1] 1766388836.316576 mapwrite 0x572000 thru 0x5756af (0x36b0 bytes) 000365[0] 1766388836.411076 mapwrite 0x27d000 thru 0x27fdb9 (0x2dba bytes) 000367[0] 1766388836.459323 read_OD 0x547000 thru 0x54ffff (0x9000 bytes) 000371[0] 1766388836.472210 read_OD 0x675000 thru 0x67afff (0x6000 bytes) 000375[1] 1766388836.479388 trunc from 0x8dc028 to 0x1ad327 (0x72ed00 bytes) 000376[1] 1766388836.504402 mapwrite 0x2a2000 thru 0x2aa182 (0x8183 bytes) 000377[1] 1766388836.553840 mapread 0x09a000 thru 0x09ed9c (0x4d9d bytes) 000380[1] 1766388836.587190 mapwrite 0x556000 thru 0x55aca1 (0x4ca2 bytes) 000386[0] 1766388836.674145 mapread 0x272000 thru 0x281203 (0xf204 bytes) 000388[1] 1766388836.696589 mapwrite 0x79d000 thru 0x7a6953 (0x9954 bytes) 000390[1] 1766388836.766366 mapread 0x33e000 thru 0x34ce10 (0xee11 bytes) 000391[0] 1766388836.827321 write_OD 0x1dc000 thru 0x1e9fff (0xe000 bytes) 000392[0] 1766388836.857304 trunc from 0x7a6954 to 0x3c6460 (0x3e04f3 bytes) 000393[1] 1766388836.873249 trunc from 0x3c6461 to 0x4d3856 (0x10d3f6 bytes) 000401[1] 1766388836.954763 read_OD 0x124000 thru 0x129fff (0x6000 bytes) 000406[0] 1766388836.968932 trunc from 0x4d3857 to 0x7ed55a (0x319d04 bytes) 000407[0] 1766388836.993013 mapwrite 0x1c1000 thru 0x1ca654 (0x9655 bytes) 000408[0] 1766388837.026785 trunc from 0x7ed55b to 0x47a150 (0x37340a bytes) 000411[0] 1766388837.045728 mapwrite 0x0ce000 thru 0x0db5b8 (0xd5b9 bytes) 000413[0] 1766388837.095186 read_OD 0x14e000 thru 0x15afff (0xd000 bytes) 000415[1] 1766388837.108418 read_OD 0x094000 thru 0x095fff (0x2000 bytes) 000416[0] 1766388837.111464 mapread 0x1b2000 thru 0x1bbaa6 (0x9aa7 bytes) 000418[1] 1766388837.122837 trunc from 0x47a151 to 0x882f37 (0x408de7 bytes) 000422[0] 1766388837.158811 mapread 0x00c000 thru 0x010661 (0x4662 bytes) 000425[0] 1766388837.186206 read_OD 0x0ec000 thru 0x0edfff (0x2000 bytes) 000426[1] 1766388837.189876 mapwrite 0x1d5000 thru 0x1e4d46 (0xfd47 bytes) 000427[1] 1766388837.232917 read_OD 0x80d000 thru 0x81bfff (0xf000 bytes) 000428[0] 1766388837.241976 mapread 0x57c000 thru 0x5858a8 (0x98a9 bytes) 000430[0] 1766388837.255535 trunc from 0x882f38 to 0x8b8d48 (0x35e11 bytes) 000431[0] 1766388837.279517 mapwrite 0x7b3000 thru 0x7ba9f8 (0x79f9 bytes) 000438[1] 1766388837.346406 write_OD 0x051000 thru 0x05afff (0xa000 bytes) 000442[0] 1766388837.376679 trunc from 0x8b8d49 to 0x16c401 (0x74c947 bytes) 000448[0] 1766388837.414075 write_OD 0x36f000 thru 0x37bfff (0xd000 bytes) 000449[0] 1766388837.446754 trunc from 0x37c000 to 0x3c0f20 (0x44f21 bytes) 000451[1] 1766388837.476314 mapread 0x2a8000 thru 0x2b12bd (0x92be bytes) 000452[0] 1766388837.496178 read_OD 0x238000 thru 0x243fff (0xc000 bytes) 000454[0] 1766388837.505294 mapread 0x2d0000 thru 0x2ddd2b (0xdd2c bytes) 000456[1] 1766388837.542056 mapread 0x299000 thru 0x29d5c8 (0x45c9 bytes) 000464[1] 1766388837.550424 mapread 0x220000 thru 0x224369 (0x436a bytes) 000465[1] 1766388837.556653 mapread 0x38c000 thru 0x3923f2 (0x63f3 bytes) 000467[1] 1766388837.572059 read_OD 0x0ea000 thru 0x0f5fff (0xc000 bytes) 000468[1] 1766388837.577486 read_OD 0x024000 thru 0x029fff (0x6000 bytes) 000469[1] 1766388837.582338 mapread 0x2eb000 thru 0x2f6269 (0xb26a bytes) 000471[1] 1766388837.613982 mapread 0x166000 thru 0x170b4e (0xab4f bytes) 000472[0] 1766388837.614884 write_OD 0x2fd000 thru 0x2fdfff (0x1000 bytes) 000473[0] 1766388837.648034 mapwrite 0x5a3000 thru 0x5ad37e (0xa37f bytes) 000474[1] 1766388837.713478 trunc from 0x5ad37f to 0x1814f9 (0x42be85 bytes) 000475[1] 1766388837.743022 mapwrite 0x47e000 thru 0x48862a (0xa62b bytes) 000477[0] 1766388837.796986 trunc from 0x48862b to 0x21cf71 (0x26b6b9 bytes) 000478[0] 1766388837.812016 trunc from 0x21cf72 to 0x513f94 (0x2f7023 bytes) 000480[1] 1766388837.832831 mapwrite 0x563000 thru 0x5656bd (0x26be bytes) 000483[0] 1766388837.890666 write_OD 0x5e8000 thru 0x5f1fff (0xa000 bytes) 000484[1] 1766388837.913119 read 0x589000 thru 0x5933bb (0xa3bc bytes) 000485[0] 1766388837.919749 read_OD 0x488000 thru 0x48afff (0x3000 bytes) 000486[1] 1766388837.925445 read 0x366000 thru 0x373209 (0xd20a bytes) 000487[0] 1766388837.931055 write_OD 0x407000 thru 0x40ffff (0x9000 bytes) 000489[1] 1766388837.955874 trunc from 0x5f2000 to 0x4c6f08 (0x12b0f7 bytes) 000490[1] 1766388837.979646 read 0x2ac000 thru 0x2b3264 (0x7265 bytes) 000492[0] 1766388837.995161 mapwrite 0x72d000 thru 0x73aad7 (0xdad8 bytes) 000494[1] 1766388838.061523 trunc from 0x73aad8 to 0x6bb736 (0x7f3a1 bytes) 000495[0] 1766388838.077941 trunc from 0x6bb737 to 0x18983c (0x531efa bytes) 000499[1] 1766388838.103487 read 0x178000 thru 0x17d92d (0x592e bytes) All operations completed A-OK! PASS 16a (19s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 02:34:00 (1766388840) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.13885 s, 60.4 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.10389 s, 80.7 MB/s Chance of close/open is 1 in 50 Seed set to 8841 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x2b1bc4 000001[1] 1766388841.454608 trunc from 00000000 to 0x2b1bc3 (0x2b1bc4 bytes) 000002[1] 1766388841.478400 trunc from 0x2b1bc4 to 0x0d5a65 (0x1dc15e bytes) 000004[1] 1766388841.498016 mapread 0x0147f7 thru 0x0164c7 (0x1cd1 bytes) 000006[0] 1766388841.520660 write 0x7ec197 thru 0x7ec864 (0x6ce bytes) 000009[0] 1766388841.549044 mapwrite 0x145f8f thru 0x1477ca (0x183c bytes) 000012[0] 1766388841.606831 write 0x410357 thru 0x411cbf (0x1969 bytes) truncating to largest ever: 0x6524fb 000015[1] 1766388841.619855 trunc from 0x7ec865 to 0x6524fa (0x19a36a bytes) 000019[1] 1766388841.655592 read 0x2ae32e thru 0x2af2a1 (0xf74 bytes) 000021[0] 1766388841.659361 write 0x3c4c39 thru 0x3c6254 (0x161c bytes) 000022[0] 1766388841.674780 mapread 0x51a152 thru 0x51af7f (0xe2e bytes) 000024[1] 1766388841.697165 read 0x27874e thru 0x279719 (0xfcc bytes) 000025[0] 1766388841.712953 mapwrite 0x104fa4 thru 0x105b8f (0xbec bytes) 000026[0] 1766388841.751347 mapread 0x59b2b5 thru 0x59ba66 (0x7b2 bytes) 000027[0] 1766388841.754636 mapwrite 0x5c22c3 thru 0x5c3f53 (0x1c91 bytes) 000028[1] 1766388841.793290 write 0x133d65 thru 0x1353e5 (0x1681 bytes) truncating to largest ever: 0x7bcd51 000029[1] 1766388841.809378 trunc from 0x6524fb to 0x7bcd50 (0x16a856 bytes) truncating to largest ever: 0x978ac2 000031[1] 1766388841.829768 trunc from 0x7bcd51 to 0x978ac1 (0x1bbd71 bytes) 000032[0] 1766388841.838432 write 0x2eaff4 thru 0x2ebcaf (0xcbc bytes) 000033[1] 1766388841.862851 read 0x94866e thru 0x94a3fe (0x1d91 bytes) 000035[0] 1766388841.892386 mapread 0x4eff4b thru 0x4f0d57 (0xe0d bytes) 000041[1] 1766388841.909477 mapread 0x3df7c4 thru 0x3e1428 (0x1c65 bytes) 000042[0] 1766388841.921115 read 0x63b4dc thru 0x63b77b (0x2a0 bytes) 000043[0] 1766388841.924850 mapread 0x406513 thru 0x406f35 (0xa23 bytes) 000046[0] 1766388841.928744 write 0x495966 thru 0x496d1b (0x13b6 bytes) 000050[1] 1766388841.956184 trunc from 0x978ac2 to 0x181d19 (0x7f6da8 bytes) 000051[0] 1766388841.982499 trunc from 0x181d1a to 0x0c6b94 (0xbb185 bytes) 000053[0] 1766388842.007874 mapwrite 0x151eea thru 0x153d3e (0x1e55 bytes) 000057[0] 1766388842.065898 read 0x0dd2d7 thru 0x0de164 (0xe8e bytes) 000059[0] 1766388842.091656 write 0x0e1c2f thru 0x0e2094 (0x466 bytes) 000061[1] 1766388842.100787 write 0x53ee71 thru 0x540687 (0x1817 bytes) 000066[0] 1766388842.140891 trunc from 0x540688 to 0x095367 (0x4ab320 bytes) 000067[0] 1766388842.155669 read 0x07618d thru 0x076af5 (0x969 bytes) 000068[1] 1766388842.156151 mapread 0x082052 thru 0x082ccb (0xc7a bytes) 000070[0] 1766388842.177107 mapwrite 0x4522df thru 0x452920 (0x642 bytes) 000072[1] 1766388842.239447 write 0x9d1bf1 thru 0x9d375e (0x1b6e bytes) 000074[1] 1766388842.264777 mapread 0x320a13 thru 0x32178f (0xd7d bytes) 000075[1] 1766388842.280439 mapwrite 0x451472 thru 0x451d7f (0x90e bytes) 000080[0] 1766388842.323879 trunc from 0x9d375f to 0x613e5f (0x3bf8ff bytes) 000085[0] 1766388842.368672 read 0x5b1a7d thru 0x5b1f2a (0x4ae bytes) 000086[1] 1766388842.379354 mapread 0x0e66f9 thru 0x0e721a (0xb22 bytes) 000087[0] 1766388842.391689 read 0x09fdef thru 0x0a134c (0x155e bytes) 000088[0] 1766388842.394493 mapread 0x016603 thru 0x016f7e (0x97c bytes) 000090[0] 1766388842.409030 mapwrite 0x14df47 thru 0x14e015 (0xcf bytes) 000091[0] 1766388842.444342 mapwrite 0x8fe456 thru 0x8fe88f (0x43a bytes) truncating to largest ever: 0x99b9ef 000092[1] 1766388842.478807 trunc from 0x8fe890 to 0x99b9ee (0x9d15f bytes) 000094[1] 1766388842.509932 read 0x55bbba thru 0x55ca30 (0xe77 bytes) 000095[0] 1766388842.523630 read 0x9131c2 thru 0x9133b1 (0x1f0 bytes) 000097[0] 1766388842.541021 mapwrite 0x9ebc4c thru 0x9ed4bd (0x1872 bytes) 000098[0] 1766388842.593129 write 0x35d2de thru 0x35e5bd (0x12e0 bytes) 000100[0] 1766388842.613357 write 0x089493 thru 0x08af7d (0x1aeb bytes) 000101[1] 1766388842.620954 trunc from 0x9ed4be to 0x382b7f (0x66a93e bytes) 000102[0] 1766388842.653768 mapread 0x37f851 thru 0x380205 (0x9b5 bytes) 000103[1] 1766388842.673877 write 0x4967ee thru 0x4972cf (0xae2 bytes) 000107[1] 1766388842.695140 write 0x191cad thru 0x191fa9 (0x2fd bytes) 000110[1] 1766388842.710472 trunc from 0x4972d0 to 0x03b46b (0x45be64 bytes) 000111[1] 1766388842.722797 mapwrite 0x87ae5b thru 0x87b1e4 (0x38a bytes) 000113[0] 1766388842.773229 trunc from 0x87b1e5 to 0x2338f4 (0x6478f0 bytes) 000114[1] 1766388842.792846 trunc from 0x2338f5 to 0x2d27e4 (0x9eef0 bytes) 000115[1] 1766388842.820757 trunc from 0x2d27e5 to 0x0f9a8c (0x1d8d58 bytes) 000117[0] 1766388842.850805 trunc from 0x0f9a8d to 0x2f954a (0x1ffabe bytes) 000118[1] 1766388842.874381 trunc from 0x2f954b to 0x5158a4 (0x21c35a bytes) 000123[1] 1766388842.903327 mapread 0x26ae6e thru 0x26c094 (0x1227 bytes) 000124[1] 1766388842.915047 trunc from 0x5158a5 to 0x2038fb (0x311fa9 bytes) 000125[1] 1766388842.927049 mapwrite 0x63040f thru 0x631f78 (0x1b6a bytes) 000131[0] 1766388843.001533 write 0x89f3e0 thru 0x89fe5d (0xa7e bytes) 000132[1] 1766388843.015780 read 0x1f6330 thru 0x1f7ae0 (0x17b1 bytes) 000134[0] 1766388843.032308 read 0x06120f thru 0x062370 (0x1162 bytes) 000139[1] 1766388843.051666 write 0x297128 thru 0x297d7f (0xc58 bytes) 000142[0] 1766388843.070148 mapwrite 0x4a9e1f thru 0x4ab90f (0x1af1 bytes) 000145[1] 1766388843.138893 mapread 0x3be54a thru 0x3bfcf4 (0x17ab bytes) 000146[1] 1766388843.158611 write 0x215ae6 thru 0x21608d (0x5a8 bytes) 000147[1] 1766388843.171330 mapwrite 0x0c01c7 thru 0x0c068e (0x4c8 bytes) 000150[1] 1766388843.219498 trunc from 0x89fe5e to 0x78536d (0x11aaf0 bytes) 000151[0] 1766388843.231697 trunc from 0x78536e to 0x2d7ef1 (0x4ad47c bytes) 000152[0] 1766388843.256662 mapread 0x22b57c thru 0x22bcb8 (0x73d bytes) 000154[1] 1766388843.261520 mapread 0x190767 thru 0x19103f (0x8d9 bytes) 000155[0] 1766388843.282498 read 0x1a753d thru 0x1a807c (0xb40 bytes) 000157[0] 1766388843.317772 mapwrite 0x15822e thru 0x159a1d (0x17f0 bytes) 000159[0] 1766388843.374252 read 0x25c289 thru 0x25d008 (0xd80 bytes) 000160[1] 1766388843.411864 write 0x11084c thru 0x111a8f (0x1244 bytes) 000161[1] 1766388843.428925 trunc from 0x2d7ef2 to 0x5f0aab (0x318bba bytes) 000162[1] 1766388843.443349 write 0x0663fa thru 0x0664ff (0x106 bytes) 000165[1] 1766388843.470766 trunc from 0x5f0aac to 0x817e2f (0x227384 bytes) 000166[0] 1766388843.480737 mapwrite 0x2b8d56 thru 0x2b8e22 (0xcd bytes) 000167[0] 1766388843.529019 trunc from 0x817e30 to 0x5edcc6 (0x22a169 bytes) 000168[1] 1766388843.540737 trunc from 0x5edcc7 to 0x537091 (0xb6c35 bytes) 000170[1] 1766388843.571085 write 0x6bcfa7 thru 0x6be4f2 (0x154c bytes) 000173[1] 1766388843.583219 read 0x54d698 thru 0x54dbe8 (0x551 bytes) 000174[0] 1766388843.586454 trunc from 0x6be4f3 to 0x1c063e (0x4fdeb4 bytes) 000175[0] 1766388843.610633 read 0x138d0b thru 0x13a92d (0x1c23 bytes) 000177[0] 1766388843.631233 mapread 0x019c1f thru 0x019eae (0x290 bytes) 000180[1] 1766388843.632427 trunc from 0x1c063f to 0x341b0b (0x1814cd bytes) 000182[1] 1766388843.657047 mapread 0x2babd7 thru 0x2bc951 (0x1d7b bytes) 000183[0] 1766388843.671527 trunc from 0x341b0c to 0x58046a (0x23e95f bytes) 000185[1] 1766388843.700767 trunc from 0x58046b to 0x6080b4 (0x87c4a bytes) 000187[1] 1766388843.714620 mapwrite 0x70f8f0 thru 0x710d42 (0x1453 bytes) 000188[1] 1766388843.742528 trunc from 0x710d43 to 0x249420 (0x4c7922 bytes) 000190[1] 1766388843.759778 mapread 0x1eeaf0 thru 0x1eef42 (0x453 bytes) 000191[1] 1766388843.763006 read 0x0b0149 thru 0x0b0ace (0x986 bytes) 000192[0] 1766388843.784394 trunc from 0x249421 to 0x22e780 (0x1aca0 bytes) 000196[0] 1766388843.831347 read 0x092f12 thru 0x093e4b (0xf3a bytes) 000198[0] 1766388843.870046 read 0x01d86f thru 0x01e95c (0x10ee bytes) 000199[1] 1766388843.878144 mapwrite 0x5f0af4 thru 0x5f0b87 (0x94 bytes) 000201[0] 1766388843.926526 mapread 0x2eff2d thru 0x2f1a02 (0x1ad6 bytes) 000202[0] 1766388843.994368 write 0x86815f thru 0x869b58 (0x19fa bytes) 000204[1] 1766388844.016172 read 0x58774b thru 0x58915a (0x1a10 bytes) 000206[0] 1766388844.030339 write 0x9b1e86 thru 0x9b39a1 (0x1b1c bytes) 000207[1] 1766388844.037778 trunc from 0x9b39a2 to 0x934257 (0x7f74a bytes) 000208[1] 1766388844.050871 mapread 0x8d68f0 thru 0x8d8222 (0x1933 bytes) 000211[0] 1766388844.077791 read 0x4770f5 thru 0x478cbc (0x1bc8 bytes) 000212[1] 1766388844.091908 mapread 0x8d4c17 thru 0x8d69c6 (0x1db0 bytes) 000213[0] 1766388844.102248 mapread 0x2f0e4b thru 0x2f2a2c (0x1be2 bytes) 000216[1] 1766388844.125983 read 0x6cfdfc thru 0x6d0473 (0x678 bytes) 000218[0] 1766388844.130067 trunc from 0x934258 to 0x6e4df1 (0x24f466 bytes) 000219[0] 1766388844.159250 read 0x5e1adf thru 0x5e28d8 (0xdfa bytes) 000220[0] 1766388844.165451 read 0x561fa2 thru 0x5634ef (0x154e bytes) 000221[1] 1766388844.168944 mapread 0x5348b3 thru 0x534da2 (0x4f0 bytes) 000222[0] 1766388844.182727 read 0x5ada7d thru 0x5ae6c1 (0xc45 bytes) 000223[1] 1766388844.195795 read 0x3b0385 thru 0x3b1f9d (0x1c19 bytes) 000224[1] 1766388844.203712 mapwrite 0x3d76e9 thru 0x3d7a9d (0x3b5 bytes) 000228[0] 1766388844.238422 read 0x64c5e5 thru 0x64d09e (0xaba bytes) 000231[1] 1766388844.242226 mapwrite 0x1644ad thru 0x164d67 (0x8bb bytes) 000234[0] 1766388844.273390 write 0x5e6c43 thru 0x5e8aef (0x1ead bytes) 000235[1] 1766388844.294193 trunc from 0x6e4df2 to 0x3e6707 (0x2fe6ea bytes) 000236[1] 1766388844.313733 read 0x107c97 thru 0x108214 (0x57e bytes) 000241[1] 1766388844.368419 mapread 0x1e252d thru 0x1e2bf3 (0x6c7 bytes) 000242[1] 1766388844.376703 mapread 0x1fd401 thru 0x1fe840 (0x1440 bytes) 000245[0] 1766388844.393668 mapwrite 0x466cee thru 0x468170 (0x1483 bytes) 000246[0] 1766388844.447945 read 0x093a45 thru 0x094046 (0x602 bytes) 000249[0] 1766388844.469796 write 0x11687c thru 0x117a96 (0x121b bytes) 000252[0] 1766388844.490089 read 0x3fdf48 thru 0x3ff5f6 (0x16af bytes) 000257[0] 1766388844.502281 trunc from 0x468171 to 0x07f926 (0x3e884a bytes) 000258[0] 1766388844.518728 mapwrite 0x4c0bf2 thru 0x4c1707 (0xb16 bytes) 000259[1] 1766388844.556639 read 0x00c433 thru 0x00c740 (0x30e bytes) 000260[0] 1766388844.569435 trunc from 0x4c1708 to 0x7bc6d4 (0x2fafcd bytes) 000263[1] 1766388844.582526 read 0x1bad2e thru 0x1bad8e (0x61 bytes) 000264[1] 1766388844.596014 write 0x0836d2 thru 0x083edd (0x80c bytes) 000266[0] 1766388844.615496 read 0x3bf534 thru 0x3c12ad (0x1d7a bytes) 000269[1] 1766388844.638530 trunc from 0x7bc6d5 to 0x9016f5 (0x145021 bytes) 000270[1] 1766388844.660198 mapread 0x65efb1 thru 0x65f229 (0x279 bytes) 000271[0] 1766388844.666724 write 0x9ea95c thru 0x9eb355 (0x9fa bytes) 000275[0] 1766388844.698522 read 0x9b6db5 thru 0x9b8bd4 (0x1e20 bytes) 000276[1] 1766388844.710252 mapwrite 0x129046 thru 0x12a945 (0x1900 bytes) 000277[0] 1766388844.751511 mapwrite 0x393834 thru 0x39495d (0x112a bytes) 000278[1] 1766388844.792670 trunc from 0x9eb356 to 0x49b4ba (0x54fe9b bytes) 000281[1] 1766388844.824528 read 0x225213 thru 0x225281 (0x6f bytes) 000283[1] 1766388844.843989 mapread 0x106de8 thru 0x107fa5 (0x11be bytes) 000286[0] 1766388844.859660 read 0x16b0c7 thru 0x16bc31 (0xb6b bytes) 000290[0] 1766388844.883898 read 0x0dad3d thru 0x0db3ff (0x6c3 bytes) 000291[1] 1766388844.886981 mapread 0x0d98ae thru 0x0d9e56 (0x5a9 bytes) 000294[0] 1766388844.906471 write 0x67ac9d thru 0x67c446 (0x17aa bytes) 000298[1] 1766388844.927481 mapread 0x55fed0 thru 0x560d21 (0xe52 bytes) 000299[1] 1766388844.943594 read 0x011142 thru 0x011e97 (0xd56 bytes) 000302[1] 1766388844.949820 read 0x2f3379 thru 0x2f5082 (0x1d0a bytes) 000304[0] 1766388844.954069 mapread 0x67c269 thru 0x67c446 (0x1de bytes) 000305[0] 1766388844.958213 mapwrite 0x7e5513 thru 0x7e69cb (0x14b9 bytes) 000307[0] 1766388845.012808 read 0x519d68 thru 0x51bb3a (0x1dd3 bytes) 000308[0] 1766388845.015729 mapwrite 0x07b48c thru 0x07c98f (0x1504 bytes) 000309[1] 1766388845.048426 trunc from 0x7e69cc to 0x29a13d (0x54c88e bytes) 000312[1] 1766388845.093047 write 0x9ca478 thru 0x9ca74c (0x2d5 bytes) 000313[1] 1766388845.096264 mapread 0x0573d7 thru 0x059260 (0x1e8a bytes) 000314[0] 1766388845.106842 read 0x790139 thru 0x791743 (0x160b bytes) 000317[0] 1766388845.141954 mapwrite 0x2ad0ab thru 0x2aea9f (0x19f5 bytes) 000319[1] 1766388845.231246 read 0x40d6c0 thru 0x40e8b6 (0x11f7 bytes) 000323[0] 1766388845.269194 write 0x66e6ca thru 0x66f025 (0x95c bytes) 000325[0] 1766388845.339661 trunc from 0x9ca74d to 0x023e7f (0x9a68cd bytes) 000327[1] 1766388845.374868 mapread 0x018c36 thru 0x018fff (0x3ca bytes) 000330[1] 1766388845.403645 mapread 0x0042a9 thru 0x00564f (0x13a7 bytes) 000331[0] 1766388845.426101 trunc from 0x023e80 to 0x782d24 (0x75eea5 bytes) 000333[1] 1766388845.475429 mapwrite 0x1c2d0a thru 0x1c2ff8 (0x2ef bytes) 000334[1] 1766388845.522387 mapread 0x160465 thru 0x16161e (0x11ba bytes) 000335[0] 1766388845.543456 trunc from 0x782d25 to 0x73a803 (0x48521 bytes) 000336[0] 1766388845.567112 read 0x4f5485 thru 0x4f612d (0xca9 bytes) 000337[1] 1766388845.581673 write 0x020d13 thru 0x02183d (0xb2b bytes) 000341[0] 1766388845.633232 trunc from 0x73a804 to 0x6991e4 (0xa161f bytes) 000343[1] 1766388845.669470 write 0x418048 thru 0x419f38 (0x1ef1 bytes) 000344[1] 1766388845.689686 mapwrite 0x081f07 thru 0x0833b7 (0x14b1 bytes) 000346[1] 1766388845.733704 trunc from 0x6991e5 to 0x70b4ad (0x722c9 bytes) 000348[1] 1766388845.747198 mapwrite 0x8d0959 thru 0x8d1969 (0x1011 bytes) 000353[0] 1766388845.815060 mapread 0x6ace33 thru 0x6aed34 (0x1f02 bytes) 000359[1] 1766388845.872692 write 0x67554a thru 0x67608f (0xb46 bytes) 000364[1] 1766388845.901886 trunc from 0x8d196a to 0x3bcdb8 (0x514bb1 bytes) 000367[1] 1766388845.924340 read 0x2a8c41 thru 0x2aa40a (0x17ca bytes) 000371[0] 1766388845.980289 mapwrite 0x470902 thru 0x47237f (0x1a7e bytes) 000372[1] 1766388846.049922 write 0x266c60 thru 0x267f2f (0x12d0 bytes) 000373[0] 1766388846.076832 mapwrite 0x06e2c2 thru 0x06f662 (0x13a1 bytes) 000374[0] 1766388846.128065 write 0x6dd5d3 thru 0x6df095 (0x1ac3 bytes) 000375[1] 1766388846.139505 read 0x573cab thru 0x575726 (0x1a7c bytes) 000378[0] 1766388846.177629 trunc from 0x6df096 to 0x412487 (0x2ccc0e bytes) 000379[1] 1766388846.204498 trunc from 0x412488 to 0x4dfc0a (0xcd783 bytes) 000380[0] 1766388846.227265 trunc from 0x4dfc0b to 0x89d874 (0x3bdc6a bytes) 000381[1] 1766388846.253175 mapwrite 0x53f9a3 thru 0x540f5f (0x15bd bytes) 000383[0] 1766388846.308876 mapread 0x6d92df thru 0x6d9ab7 (0x7d9 bytes) 000384[1] 1766388846.322454 mapread 0x67b242 thru 0x67c875 (0x1634 bytes) 000388[0] 1766388846.341410 read 0x125a2c thru 0x125a89 (0x5e bytes) 000391[1] 1766388846.355342 trunc from 0x89d875 to 0x386b44 (0x516d30 bytes) 000394[1] 1766388846.386177 trunc from 0x386b45 to 0x183578 (0x2035cc bytes) 000395[1] 1766388846.404358 write 0x333ed9 thru 0x33529b (0x13c3 bytes) 000397[0] 1766388846.424694 trunc from 0x33529c to 0x64539f (0x310104 bytes) 000399[1] 1766388846.448912 trunc from 0x6453a0 to 0x0dc9c6 (0x5689d9 bytes) 000400[0] 1766388846.484402 read 0x0d55ed thru 0x0d62c8 (0xcdc bytes) 000401[0] 1766388846.513788 read 0x0d487d thru 0x0d5741 (0xec5 bytes) 000408[1] 1766388846.530965 mapwrite 0x581662 thru 0x581e27 (0x7c6 bytes) 000409[0] 1766388846.583436 mapwrite 0x1a81a7 thru 0x1a8dfd (0xc57 bytes) 000416[0] 1766388846.723186 read 0x3147f3 thru 0x314a2d (0x23b bytes) 000417[0] 1766388846.733311 mapread 0x4de99e thru 0x4dfb11 (0x1174 bytes) 000418[0] 1766388846.751307 mapread 0x35b497 thru 0x35c882 (0x13ec bytes) 000420[0] 1766388846.752373 mapwrite 0x121654 thru 0x1224db (0xe88 bytes) 000421[0] 1766388846.777165 mapwrite 0x1dcae1 thru 0x1ddfa5 (0x14c5 bytes) 000423[1] 1766388846.812326 mapwrite 0x9cae4d thru 0x9cb1a2 (0x356 bytes) 000425[1] 1766388846.878888 mapread 0x501aba thru 0x5020a7 (0x5ee bytes) 000426[1] 1766388846.901918 write 0x0e8568 thru 0x0e8d4a (0x7e3 bytes) 000428[0] 1766388846.915186 mapread 0x838a62 thru 0x839b66 (0x1105 bytes) 000435[0] 1766388846.963974 mapwrite 0x083333 thru 0x083de8 (0xab6 bytes) 000436[1] 1766388847.005923 mapwrite 0x9f048d thru 0x9f06f3 (0x267 bytes) 000440[1] 1766388847.057396 read 0x79cec7 thru 0x79d87b (0x9b5 bytes) 000442[0] 1766388847.069484 read 0x248739 thru 0x248d9e (0x666 bytes) 000443[1] 1766388847.079787 trunc from 0x9f06f4 to 0x3fcce5 (0x5f3a0e bytes) 000445[0] 1766388847.105650 mapread 0x263b5d thru 0x265317 (0x17bb bytes) 000447[1] 1766388847.138932 write 0x91e7a1 thru 0x91ed30 (0x590 bytes) 000448[1] 1766388847.153102 mapwrite 0x6ee864 thru 0x6efd44 (0x14e1 bytes) 000451[1] 1766388847.191243 trunc from 0x91ed31 to 0x5cc985 (0x3523ab bytes) 000452[1] 1766388847.209137 mapwrite 0x9c67e2 thru 0x9c699b (0x1ba bytes) 000455[0] 1766388847.253520 mapread 0x286fda thru 0x28854d (0x1574 bytes) 000456[1] 1766388847.272684 mapwrite 0x038254 thru 0x03924e (0xffb bytes) 000459[1] 1766388847.304908 read 0x84841d thru 0x848d21 (0x905 bytes) 000462[1] 1766388847.329029 write 0x40f1eb thru 0x4106b1 (0x14c7 bytes) 000464[0] 1766388847.356124 trunc from 0x9c699c to 0x13a1d4 (0x88c7c7 bytes) 000466[1] 1766388847.399654 trunc from 0x13a1d5 to 0x8e24d5 (0x7a8301 bytes) 000479[1] 1766388847.458818 write 0x410e79 thru 0x412a33 (0x1bbb bytes) 000480[0] 1766388847.471347 mapwrite 0x100244 thru 0x101d23 (0x1ae0 bytes) 000481[0] 1766388847.508730 mapread 0x6904d6 thru 0x691e19 (0x1944 bytes) 000482[1] 1766388847.524844 mapread 0x1a87b5 thru 0x1a93be (0xc0a bytes) 000484[1] 1766388847.538778 read 0x6ea1e7 thru 0x6ead5b (0xb75 bytes) 000486[0] 1766388847.542246 mapread 0x5a7c53 thru 0x5a9724 (0x1ad2 bytes) 000488[0] 1766388847.547539 read 0x889136 thru 0x889426 (0x2f1 bytes) 000490[1] 1766388847.551582 mapwrite 0x5b8092 thru 0x5b8a12 (0x981 bytes) 000491[0] 1766388847.582265 write 0x9d1a98 thru 0x9d394a (0x1eb3 bytes) 000492[1] 1766388847.594274 write 0x3f0834 thru 0x3f0e43 (0x610 bytes) 000494[1] 1766388847.617325 trunc from 0x9d394b to 0x0736ca (0x960280 bytes) 000495[1] 1766388847.627808 write 0x84a299 thru 0x84a9e6 (0x74e bytes) 000497[1] 1766388847.653941 write 0x62a397 thru 0x62b8c7 (0x1531 bytes) 000499[1] 1766388847.678419 mapwrite 0x3ef29d thru 0x3effff (0xd63 bytes) All operations completed A-OK! PASS 16b (9s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 02:34:09 (1766388849) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (2s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 02:34:11 (1766388851) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.136106 s, 61.6 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.124577 s, 67.3 MB/s ldlm.namespaces.MGC192.168.203.124@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8cd00388b800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8cd018895000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8cd00388b800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8cd018895000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8cd00388b800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8cd018895000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.34525 s, 31.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.245 s, 84.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.04027 s, 101 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.07412 s, 50.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.83538 s, 37.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.48671 s, 70.5 MB/s PASS 16d (17s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 02:34:28 (1766388868) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.119838 s, 87.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489187 s, 21.4 MB/s PASS 16e (3s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 02:34:31 (1766388871) TIMEOUT PASS 16f (23s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 02:34:54 (1766388894) TIMEOUT PASS 16g (24s) == sanityn test 16h: mmap read after truncate file ======= 02:35:18 (1766388918) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (3s) == sanityn test 16i: read after truncate file ============ 02:35:21 (1766388921) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (3s) == sanityn test 16j: race dio with buffered i/o ========== 02:35:24 (1766388924) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.904269 s, 29.0 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.505613 s, 51.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.939236 s, 27.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.617118 s, 42.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.760135 s, 34.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.723059 s, 36.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.924451 s, 28.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.838152 s, 31.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.850712 s, 30.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.931909 s, 28.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.998504 s, 26.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.806156 s, 32.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.843943 s, 31.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.02259 s, 25.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.08882 s, 24.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.809321 s, 32.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.945574 s, 27.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.20706 s, 21.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.27115 s, 20.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (15s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 02:35:39 (1766388939) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: skipping zero size read test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: fsx: Operation not supported test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: skipping zero size read test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x32cce75 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Mon Dec 22 02:35:53 EST 2025 + fsx 2 done: rc=0 Mon Dec 22 02:35:53 EST 2025 + fsx 3 done: rc=0 Mon Dec 22 02:35:53 EST 2025 All operations completed A-OK! All operations completed A-OK! + fsx 4 done: rc=0 Mon Dec 22 02:35:53 EST 2025 + fsx 5 done: rc=0 Mon Dec 22 02:35:53 EST 2025 PASS 16k (17s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 02:35:56 (1766388956) fail_loc=0x8000030a PASS 17 (5s) == sanityn test 18: mmap sanity check =========================================================================================== 02:36:01 (1766388961) mmap test1: basic mmap operation (PASS, 0.050681s) mmap test2: MAP_PRIVATE not write back (PASS, 0.03445s) mmap test3: concurrent mmap ops on two nodes (PASS, 9.9492s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.1759s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.060975s) mmap test6: check mmap write/read content on two nodes (PASS, 0.080189s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 0.50106s) mmap test11: multi-thread shared mmap (PASS, 0.50512s) mmap test10: mtime not change for readonly mmap access (PASS, 1.0253s) PASS 18 (20s) == sanityn test 19: test concurrent uncached read races ========================================================================= 02:36:21 (1766388981) oleg324-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (2s) == sanityn test 20: test extra readahead page left in cache ============================================================== 02:36:23 (1766388983) PASS 20 (3s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 02:36:26 (1766388986) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (2s) == sanityn test 23: others should see updated atime while another read============================================================== 02:36:28 (1766388988) now is 1766388989 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7552 new atime is 1766389050 PASS 23 (64s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 02:37:32 (1766389052) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.6K 388 490.2K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 402 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 398 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 388 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.6K 388 490.2K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 402 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 398 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 388 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 3.2M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 6.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502379 388 501991 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118066 402 117664 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117230 398 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234884 388 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.6K 388 490.2K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 402 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 398 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 388 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502379 388 501991 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118066 402 117664 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117230 398 116832 1% /mnt/lustre[OST:1] filesystem_summary: 234884 388 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502379 388 501991 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118066 402 117664 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117230 398 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234884 388 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (3s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 02:37:35 (1766389055) PASS 24b (3s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 02:37:38 (1766389058) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (2s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 02:37:40 (1766389060) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (2s) == sanityn test 26a: allow mtime to get older ============ 02:37:42 (1766389062) utime: good mknod times 1766389061 <= 1766389062 <= 1766389062 for /mnt/lustre/f26a utime: good mknod times 1766389061 <= 1766389062 <= 1766389062 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (3s) == sanityn test 26b: sync mtime between ost and mds ====== 02:37:45 (1766389065) PASS 26b (4s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 02:37:50 (1766389070) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7552 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2025-12-22 02:37:50.000000000 -0500 Birth: 2025-12-22 02:37:50.000000000 -0500 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2025-12-22 02:37:50.000000000 -0500 Birth: 2025-12-22 02:37:50.000000000 -0500 PASS 26c (4s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 02:37:53 (1766389073) 0 PASS 30 (5s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 02:37:58 (1766389078) fail_loc=0x314 PASS 31a (3s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 02:38:01 (1766389081) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg324-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.124@tcp:/lustre 7542784 39936 7498752 1% /mnt/lustre2 PASS 31b (13s) == sanityn test 31r: open-rename(replace) race =========== 02:38:14 (1766389094) -rw-r--r-- 1 root root 0 Dec 22 02:38 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (6s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 02:38:20 (1766389100) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1207: 42819 Terminated tail -f $tfile PASS 31s (3s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 02:38:23 (1766389103) PASS 31t (3s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 02:38:27 (1766389107) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (1s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 02:38:28 (1766389108) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (2s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 02:38:30 (1766389110) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (1s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 02:38:31 (1766389111) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (1s) == sanityn test 34: no lock timeout under IO ============= 02:38:32 (1766389112) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg324-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8cd00388b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8cd00388b800.ost_server_uuid in FULL state after 0 sec oleg324-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8cd00388b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8cd00388b800.ost_server_uuid in IDLE state after 0 sec oleg324-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8cd00388b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8cd00388b800.ost_server_uuid in FULL state after 0 sec oleg324-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8cd00388b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8cd00388b800.ost_server_uuid in IDLE state after 0 sec oleg324-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8cd00388b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8cd00388b800.ost_server_uuid in IDLE state after 0 sec oleg324-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8cd00388b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8cd00388b800.ost_server_uuid in IDLE state after 0 sec PASS 34 (98s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 02:40:10 (1766389210) fail_loc=0x80000317 Race attempt 0 Wait for 48937 49062 for 60 sec... - open/close 1489 (time 1766389223.10 total 10.00 last 148.85) - open/close 3753 (time 1766389233.10 total 20.01 last 226.25) total: 4000 open/close in 21.29 seconds: 187.88 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.203.124@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre 192.168.203.124@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre2 PASS 35 (66s) == sanityn test 36: handle ESTALE/open-unlink correctly == 02:41:16 (1766389276) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg324-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.707549 s, 74.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7552 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7497728) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.61505 s, 32.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7552 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7490560) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.53809 s, 34.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7552 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7493632) after_dd(7444480) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.00114 s, 52.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7552 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7497728) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.821921 s, 63.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7552 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7495680) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.862708 s, 60.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7552 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7493632) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.704718 s, 74.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7552 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7493632) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.709559 s, 73.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7552 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7493632) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.903235 s, 58.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7552 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7493632) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.784726 s, 66.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7552 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7491584) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.799581 s, 65.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7552 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7493632) after_dd(7440384) after(7491584) PASS 36 (295s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 02:46:11 (1766389571) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7552 - create 1516 (time 1766389584.58 total 10.00 last 151.55) - create 3027 (time 1766389594.58 total 20.01 last 151.07) - create 4509 (time 1766389604.59 total 30.01 last 148.08) - create 5960 (time 1766389614.59 total 40.02 last 144.99) - create 7452 (time 1766389624.60 total 50.02 last 149.19) - create 8891 (time 1766389634.60 total 60.02 last 143.85) total: 10000 create in 67.05 seconds: 149.14 ops/second PASS 37 (75s) == sanityn test 39a: file mtime does not change after rename ========================================================== 02:47:26 (1766389646) repeat after cancel_lru_locks PASS 39a (5s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 02:47:31 (1766389651) repeat after cancel_lru_locks PASS 39b (6s) == sanityn test 39c: check truncate mtime update ================================================================================ 02:47:37 (1766389657) repeat after cancel_lru_locks PASS 39c (6s) == sanityn test 39d: sync write should update mtime ====== 02:47:43 (1766389663) fail_loc=0x411 fail_loc=0 PASS 39d (4s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 02:47:49 (1766389669) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (14s) == sanityn test 40c: pdirops: link and others ======================================================================== 02:48:03 (1766389683) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (14s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 02:48:17 (1766389697) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (20s) == sanityn test 40e: pdirops: rename and others ======================================================================== 02:48:37 (1766389717) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (19s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 02:48:56 (1766389736) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (18s) == sanityn test 41b: pdirops: create vs create ======================================================================== 02:49:14 (1766389754) open: File exists Conflict PASS 41b (14s) == sanityn test 41c: pdirops: create vs link ======================================================================== 02:49:28 (1766389768) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (14s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 02:49:42 (1766389782) Conflict PASS 41d (13s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 02:49:55 (1766389795) Conflict PASS 41e (13s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 02:50:08 (1766389808) Conflict PASS 41f (14s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 02:50:22 (1766389822) Conflict PASS 41g (14s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 02:50:36 (1766389836) Conflict PASS 41h (14s) == sanityn test 41i: reint_open: create vs create ======== 02:50:50 (1766389850) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (942s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 03:06:32 (1766390792) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (8s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 03:06:40 (1766390800) open: File exists Conflict PASS 42b (7s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 03:06:47 (1766390807) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (7s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 03:06:54 (1766390814) Conflict PASS 42d (7s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 03:07:01 (1766390821) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (7s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 03:07:09 (1766390829) Conflict PASS 42f (7s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 03:07:16 (1766390836) Conflict PASS 42g (7s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 03:07:23 (1766390843) Conflict PASS 42h (7s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 03:07:30 (1766390850) PASS 43a (35s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 03:08:05 (1766390885) Conflict PASS 43b (7s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 03:08:12 (1766390892) Conflict PASS 43c (8s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 03:08:20 (1766390900) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (7s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 03:08:27 (1766390907) Conflict PASS 43e (7s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 03:08:34 (1766390914) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (7s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 03:08:41 (1766390921) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (8s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 03:08:49 (1766390929) Conflict PASS 43h (9s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 03:08:58 (1766390938) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (1s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 03:09:00 (1766390940) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists PASS 43j (67s) == sanityn test 43k: unlink vs create ==================== 03:10:07 (1766391007) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (642s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 03:20:49 (1766391649) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Mon Dec 22 03:20:56 EST 2025 PASS 44a (10s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 03:20:59 (1766391659) open: File exists Conflict PASS 44b (10s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 03:21:10 (1766391670) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (10s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 03:21:20 (1766391680) Conflict PASS 44d (10s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 03:21:30 (1766391690) Conflict PASS 44e (9s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 03:21:39 (1766391699) Conflict PASS 44f (11s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 03:21:50 (1766391710) Conflict PASS 44g (11s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 03:22:01 (1766391721) Conflict PASS 44h (12s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 03:22:13 (1766391733) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (2s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 03:22:16 (1766391736) PASS 45a (105s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 03:24:00 (1766391840) Conflict PASS 45b (11s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 03:24:11 (1766391851) Conflict PASS 45c (11s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 03:24:22 (1766391862) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (10s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 03:24:32 (1766391872) Conflict PASS 45e (11s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 03:24:43 (1766391883) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (11s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 03:24:55 (1766391895) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (10s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 03:25:05 (1766391905) Conflict PASS 45h (11s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 03:25:16 (1766391916) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (2s) == sanityn test 45j: read vs rename ====================== 03:25:18 (1766391918) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (825s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 03:39:03 (1766392743) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (7s) == sanityn test 46b: pdirops: link vs create ======================================================================== 03:39:10 (1766392750) open: File exists Conflict PASS 46b (8s) == sanityn test 46c: pdirops: link vs link ======================================================================== 03:39:18 (1766392758) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (8s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 03:39:26 (1766392766) Conflict PASS 46d (6s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 03:39:32 (1766392772) Conflict PASS 46e (7s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 03:39:39 (1766392779) Conflict PASS 46f (7s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 03:39:46 (1766392786) Conflict PASS 46g (7s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 03:39:53 (1766392793) Conflict PASS 46h (9s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 03:40:02 (1766392802) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (1s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 03:40:04 (1766392804) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (1s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 03:40:05 (1766392805) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (2s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 03:40:07 (1766392807) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (1s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 03:40:08 (1766392808) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (2s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 03:40:10 (1766392810) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (1s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 03:40:12 (1766392812) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (1s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 03:40:13 (1766392813) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (2s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 03:40:15 (1766392815) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.0405824 s, 252 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (8s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 03:40:23 (1766392823) 0+1 records in 0+1 records out 158 bytes copied, 0.0114784 s, 13.8 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3510: kill: (260177) - No such process PASS 51a (5s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 03:40:28 (1766392828) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00567546 s, 180 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279728 s, 37.5 MB/s 3145728 PASS 51b (16s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 03:40:44 (1766392844) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0161105 s, 63.6 kB/s PASS 51c (8s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 03:40:52 (1766392852) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253196 s, 41.4 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (4s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 03:40:57 (1766392857) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2451 0x993 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2451 0x993 0x240000400 read lease(1) has applied. PASS 51e (6s) == sanityn test 54: rename locking ======================= 03:41:02 (1766392862) llite.lustre-ffff8cd00388b800.xattr_cache=1 llite.lustre-ffff8cd018895000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505011 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 03:41:03.000000000 -0500 Modify: 2025-12-22 03:41:03.000000000 -0500 Change: 2025-12-22 03:41:03.000000000 -0500 Birth: 2025-12-22 03:41:03.000000000 -0500 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505014 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 03:41:09.000000000 -0500 Modify: 2025-12-22 03:41:09.000000000 -0500 Change: 2025-12-22 03:41:09.000000000 -0500 Birth: 2025-12-22 03:41:09.000000000 -0500 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505017 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 03:41:14.000000000 -0500 Modify: 2025-12-22 03:41:14.000000000 -0500 Change: 2025-12-22 03:41:14.000000000 -0500 Birth: 2025-12-22 03:41:15.000000000 -0500 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505016 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 03:41:14.000000000 -0500 Modify: 2025-12-22 03:41:15.000000000 -0500 Change: 2025-12-22 03:41:15.000000000 -0500 Birth: 2025-12-22 03:41:15.000000000 -0500 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505019 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 03:41:20.000000000 -0500 Modify: 2025-12-22 03:41:21.000000000 -0500 Change: 2025-12-22 03:41:21.000000000 -0500 Birth: 2025-12-22 03:41:21.000000000 -0500 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 03:41:20.000000000 -0500 Modify: 2025-12-22 03:41:21.000000000 -0500 Change: 2025-12-22 03:41:21.000000000 -0500 Birth: 2025-12-22 03:41:21.000000000 -0500 PASS 54 (26s) == sanityn test 55a: rename vs unlink target dir ========= 03:41:28 (1766392888) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (9s) == sanityn test 55b: rename vs unlink source dir ========= 03:41:37 (1766392897) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (8s) == sanityn test 55c: rename vs unlink orphan target dir == 03:41:45 (1766392905) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (14s) == sanityn test 55d: rename file vs link ================= 03:41:59 (1766392919) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (11s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 03:42:10 (1766392930) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (2s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 03:42:12 (1766392932) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (14s) == sanityn test 55g: rename: race with trylock =========== 03:42:26 (1766392946) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre/d55g.sanityn/a1/a2' to a subdirectory of itself, '/mnt/lustre/d55g.sanityn/b1/b2/b3/b4/a2' We survived race with trylock test fail_loc=0x0 PASS 55g (15s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 03:42:41 (1766392961) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1766392961 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 1.19684 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (53s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 03:43:34 (1766393014) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1766393016 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 2.83894 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (49s) == sanityn test 60: Verify data_version behaviour ======== 03:44:23 (1766393063) PASS 60 (3s) == sanityn test 70a: cd directory && rm directory ======== 03:44:26 (1766393066) PASS 70a (3s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 03:44:29 (1766393069) PASS 70b (4s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 03:44:33 (1766393073) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00799562 s, 8.2 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (1s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 03:44:34 (1766393074) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00448739 s, 14.6 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (2s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 03:44:36 (1766393076) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (1s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 03:44:37 (1766393077) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00519596 s, 12.6 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (1s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 03:44:38 (1766393078) llite.lustre-ffff8cd00388b800.xattr_cache=1 llite.lustre-ffff8cd018895000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (3s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 03:44:41 (1766393081) llite.lustre-ffff8cd00388b800.xattr_cache=1 llite.lustre-ffff8cd018895000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (3s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 03:44:44 (1766393084) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00597482 s, 171 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00610529 s, 168 kB/s 272954: get lock2 272955: get lock1 272955: done 272955: get lock2 272954: done 272954: get lock1 fcntl cmd 7 failed: Resource deadlock avoided 272954: cannot get lock1: Resource deadlock avoided 272955: done 272955: exit rc=0 272954: done 272954: exit rc=0 PASS 74 (7s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 03:44:51 (1766393091) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0243714 s, 86.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154488 s, 67.9 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4193: 273537 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (8s) == sanityn test 76: Verify MDT open_files listing ======== 03:44:59 (1766393099) opening files: ........ closing files: ........ PASS 76 (27s) == sanityn test 77a: check FIFO NRS policy =============== 03:45:26 (1766393126) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.172442 s, 97.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632627 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054258 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629182 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715779 s, 14.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813927 s, 12.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776596 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721739 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610657 s, 17.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916628 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990014 s, 10.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873315 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959948 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844804 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854647 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108604 s, 9.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8cd00388b800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8cd00388b800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925765 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198968 s, 52.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303498 s, 34.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254743 s, 41.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277133 s, 37.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023803 s, 44.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257044 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257707 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031105 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119834 s, 87.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305479 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00547876 s, 191 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361615 s, 29.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112106 s, 93.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312177 s, 33.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326778 s, 32.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8cd00388b800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192486 s, 54.5 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg324-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg324-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (4s) == sanityn test 77b: check CRR-N NRS policy ============== 03:45:30 (1766393130) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.186413 s, 90.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235214 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381991 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328419 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564932 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606404 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706096 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985914 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113625 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10949 s, 9.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127594 s, 8.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118445 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947462 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116902 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12715 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135433 s, 7.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8cd00388b800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8cd00388b800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133168 s, 7.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00433726 s, 242 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162166 s, 64.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012578 s, 83.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256655 s, 40.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00692581 s, 151 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210969 s, 49.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00466349 s, 225 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189383 s, 55.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00422456 s, 248 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00456254 s, 230 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00332377 s, 315 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00435876 s, 241 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00391797 s, 268 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00452974 s, 231 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00634998 s, 165 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00279581 s, 375 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg324-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg324-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.149494 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369443 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463266 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694849 s, 15.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8cd00388b800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771351 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095687 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133565 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134378 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123446 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13207 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125601 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144298 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157584 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16121 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175608 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146756 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152982 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00757662 s, 138 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144666 s, 72.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101522 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00479047 s, 219 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198379 s, 52.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279517 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287828 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00310872 s, 337 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00546219 s, 192 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0055421 s, 189 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00664474 s, 158 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053351 s, 19.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187931 s, 55.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141407 s, 74.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182414 s, 57.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00617323 s, 170 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg324-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg324-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (5s) == sanityn test 77c: check ORR NRS policy ================ 03:45:35 (1766393135) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.219802 s, 76.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041162 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387756 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114248 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136483 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166976 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173233 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149798 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169093 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181964 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19655 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167425 s, 6.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183299 s, 5.7 MB/sldlm.namespaces.lustre-OST0000-osc-ffff8cd00388b800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8cd00388b800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186585 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202725 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175352 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209644 s, 5.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231342 s, 45.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306209 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155903 s, 67.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161391 s, 65.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162451 s, 64.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198871 s, 52.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218093 s, 48.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218003 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219573 s, 47.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209756 s, 50.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319151 s, 32.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251622 s, 41.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272023 s, 38.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273106 s, 38.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279779 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294957 s, 35.6 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg324-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg324-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.172842 s, 97.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417697 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466695 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964715 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700873 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935014 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103599 s, 10.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8cd00388b800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112652 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114099 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117604 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114672 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929395 s, 11.3 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129302 s, 8.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116541 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122767 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988926 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1348 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00707493 s, 148 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357349 s, 29.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297193 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190808 s, 55.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205351 s, 51.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351202 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249246 s, 42.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297286 s, 35.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267297 s, 39.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313413 s, 33.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282639 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403339 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339841 s, 30.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329307 s, 31.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331143 s, 31.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8cd00388b800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00914412 s, 115 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg324-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg324-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.195836 s, 85.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028636 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346261 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975834 s, 10.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8cd00388b800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104331 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972691 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0974781 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975841 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106709 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121629 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13625 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141075 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143279 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142006 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164843 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158154 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170329 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00544473 s, 193 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108318 s, 96.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023141 s, 45.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252268 s, 41.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291813 s, 35.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166519 s, 63.0 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342655 s, 30.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231317 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173323 s, 60.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265445 s, 39.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313739 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208965 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283712 s, 37.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8cd00388b800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8cd00388b800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00647429 s, 162 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025019 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264484 s, 39.6 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg324-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg324-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (6s) == sanityn test 77d: check TRR nrs policy ================ 03:45:41 (1766393141) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.160075 s, 105 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408537 s, 25.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047547 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389649 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581031 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589047 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784464 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908931 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709561 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962207 s, 10.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844384 s, 12.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088744 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102836 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874578 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121768 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102986 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975814 s, 10.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147048 s, 71.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00682056 s, 154 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252178 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166715 s, 62.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159713 s, 65.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00468826 s, 224 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025485 s, 41.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157015 s, 66.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193454 s, 54.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285358 s, 36.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243199 s, 43.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273288 s, 38.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232442 s, 45.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8cd00388b800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00538123 s, 195 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00766759 s, 137 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00332888 s, 315 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg324-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg324-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.204095 s, 82.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038306 s, 27.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039269 s, 26.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509764 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625251 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466355 s, 22.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745123 s, 14.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671514 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769256 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979323 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661049 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785108 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629477 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114063 s, 9.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8cd00388b800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8cd00388b800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1036 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109144 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108643 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029802 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286015 s, 36.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420214 s, 25.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192415 s, 54.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145027 s, 72.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398959 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304285 s, 34.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351972 s, 29.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139375 s, 75.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370712 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163661 s, 64.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360734 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327452 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036433 s, 28.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8cd00388b800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff8cd00388b800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00842671 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241943 s, 43.3 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg324-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg324-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.168645 s, 99.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229034 s, 45.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603742 s, 17.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083887 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918522 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102895 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942161 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859424 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106686 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122389 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159882 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181089 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127889 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166171 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168615 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148408 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142226 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137476 s, 76.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210213 s, 49.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017929 s, 58.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177505 s, 59.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182264 s, 57.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143375 s, 73.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238015 s, 44.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256813 s, 40.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274129 s, 38.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00830236 s, 126 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116761 s, 89.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226193 s, 46.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291904 s, 35.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00879754 s, 119 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00334996 s, 313 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8cd00388b800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8cd00388b800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104159 s, 101 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg324-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg324-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (7s) == sanityn test 77e: check TBF NID nrs policy ============ 03:45:48 (1766393148) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.203.24@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.199067 s, 84.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298448 s, 35.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571063 s, 18.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428489 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817297 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100599 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111564 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116411 s, 9.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8cd00388b800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12517 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14179 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151151 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143855 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163536 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166128 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154513 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136967 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150272 s, 7.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8cd00388b800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8cd00388b800.lock_unused_count=4 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10332 s, 10.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107119 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117946 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107934 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11954 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109583 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123199 s, 8.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114346 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115621 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11555 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112307 s, 9.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117877 s, 8.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115771 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115139 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118953 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121761 s, 8.6 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg324-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg324-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.181815 s, 92.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344735 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265015 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822624 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845651 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118296 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104882 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125905 s, 8.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8cd00388b800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8cd00388b800.lock_unused_count=4 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123313 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105461 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135902 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118643 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132806 s, 7.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137655 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119242 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139956 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131241 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186678 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173858 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174387 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174232 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188017 s, 5.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180251 s, 5.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179481 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180155 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184072 s, 5.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180323 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194373 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183139 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184503 s, 5.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195399 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186963 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18926 s, 5.5 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg324-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg324-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.189032 s, 88.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220916 s, 47.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360015 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415189 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863021 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100544 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889392 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112537 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126225 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122634 s, 8.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147644 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134771 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139936 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151819 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156361 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147383 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189631 s, 5.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8cd00388b800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8cd00388b800.lock_unused_count=5 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01771 s, 59.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179169 s, 58.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347666 s, 30.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274468 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203535 s, 51.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265206 s, 39.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339188 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226161 s, 46.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373647 s, 28.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359953 s, 29.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462599 s, 22.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041359 s, 25.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425366 s, 24.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371512 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273417 s, 38.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339064 s, 30.9 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg324-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg324-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.174715 s, 96.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392941 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361336 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357286 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746596 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504485 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661014 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622993 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883871 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578973 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104547 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155436 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145444 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337275 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16288 s, 6.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247903 s, 42.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447421 s, 23.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170989 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182144 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171239 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157554 s, 66.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299852 s, 35.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223573 s, 46.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139971 s, 74.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254813 s, 41.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00643108 s, 163 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135313 s, 77.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00340374 s, 308 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011038 s, 95.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347621 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017597 s, 59.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8cd00388b800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8cd00388b800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222706 s, 47.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00713895 s, 147 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg324-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg324-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (10s) == sanityn test 77f: check TBF JobID nrs policy ========== 03:45:58 (1766393158) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg324-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.203189 s, 82.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418491 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532287 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401554 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564654 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457521 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565467 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472457 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551658 s, 19.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130518 s, 8.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130033 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146039 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125141 s, 8.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147824 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145442 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146914 s, 7.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8cd00388b800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff8cd00388b800.lock_unused_count=3 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240468 s, 4.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8cd00388b800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191054 s, 54.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405533 s, 25.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435775 s, 24.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277592 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034212 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405792 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214543 s, 48.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292908 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235425 s, 44.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216737 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198803 s, 52.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00886945 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181711 s, 57.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167803 s, 62.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168599 s, 62.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160513 s, 65.3 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg324-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg324-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg324-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.177554 s, 94.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374931 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328706 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072045 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979112 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970647 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122872 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115184 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139459 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160621 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142827 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151957 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17062 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164733 s, 6.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163923 s, 6.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167117 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165496 s, 6.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8cd00388b800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8cd00388b800.lock_unused_count=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256715 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338758 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212361 s, 49.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308274 s, 34.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028456 s, 36.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317549 s, 33.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353362 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295051 s, 35.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122904 s, 85.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172177 s, 60.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382087 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336203 s, 31.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110947 s, 94.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352865 s, 29.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161652 s, 64.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00975742 s, 107 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg324-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg324-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg324-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.203746 s, 82.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544057 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625922 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570087 s, 18.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103349 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643446 s, 16.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741145 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078957 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868397 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739003 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160222 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167184 s, 6.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups:1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175511 s, 6.0 MB/s 500 [declare] [-a] [pids_r] ldlm.namespaces.lustre-OST0001-osc-ffff8cd00388b800.lock_unused_count=3 execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174169 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191801 s, 5.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191191 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208722 s, 5.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373016 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034861 s, 30.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328913 s, 31.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334258 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345592 s, 30.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033623 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344673 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355464 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430798 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401273 s, 26.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381831 s, 27.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407617 s, 25.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395964 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379531 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569573 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487582 s, 21.5 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg324-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg324-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg324-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.165361 s, 101 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466418 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835872 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095168 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120058 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122291 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136629 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111074 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135638 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137793 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131649 s, 8.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122827 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139451 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148373 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133315 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131916 s, 7.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132753 s, 7.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162068 s, 64.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211093 s, 49.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189136 s, 55.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030186 s, 34.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302598 s, 34.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199887 s, 52.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0084234 s, 124 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280701 s, 37.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238699 s, 43.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153541 s, 68.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339544 s, 30.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375072 s, 28.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112197 s, 93.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018719 s, 56.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212549 s, 49.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217946 s, 48.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8cd00388b800.lock_unused_count=4 removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg324-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg324-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (10s) == sanityn test 77g: Change TBF type directly ============ 03:46:08 (1766393168) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (4s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 03:46:13 (1766393173) oleg324-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg324-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 oleg324-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg324-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg324-client: oleg324-server: ssh exited with exit code 22 oleg324-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg324-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg324-client: oleg324-server: ssh exited with exit code 22 oleg324-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg324-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg324-client: oleg324-server: ssh exited with exit code 22 PASS 77h (5s) == sanityn test 77i: Change rank of TBF rule ============= 03:46:18 (1766393178) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (10s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 03:46:28 (1766393188) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.286995 s, 58.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441711 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03544 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107988 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115169 s, 9.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127138 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906327 s, 11.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1047 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102666 s, 10.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108085 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114844 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105752 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128593 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217955 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204214 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270704 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262843 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205097 s, 51.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292534 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030195 s, 34.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344697 s, 30.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321212 s, 32.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350931 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233451 s, 44.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264385 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278147 s, 37.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303311 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329694 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029214 s, 35.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357771 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02944 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360678 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384766 s, 27.3 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg324-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg324-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.04272 s, 20.8 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.489 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (56s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 03:47:24 (1766393244) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg324-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg324-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg324-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg324-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg324-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.24701 s, 13.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147494 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149263 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140683 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120622 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135535 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162595 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17961 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159199 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135205 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166739 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154063 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180169 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185697 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157589 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215393 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228658 s, 4.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303984 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331419 s, 31.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300644 s, 34.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310913 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355089 s, 29.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036079 s, 29.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036213 s, 29.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398835 s, 26.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291751 s, 35.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400246 s, 26.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302716 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311553 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367593 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365376 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330618 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428747 s, 24.5 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg324-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg324-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7789 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9759 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg324-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg324-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg324-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg324-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg324-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.23938 s, 13.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101975 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104237 s, 10.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117717 s, 8.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141726 s, 7.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13958 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140444 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148748 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149259 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165886 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176982 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156772 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174134 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160091 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159503 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171289 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171717 s, 6.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218662 s, 48.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276727 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322441 s, 32.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331093 s, 31.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330085 s, 31.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268354 s, 39.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252312 s, 41.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290481 s, 36.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372346 s, 28.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298988 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311606 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277747 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265222 s, 39.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276646 s, 37.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346888 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312713 s, 33.5 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg324-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg324-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8507 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9464 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (137s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 03:49:41 (1766393381) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg324-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg324-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg324-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg324-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg324-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.24645 s, 13.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477417 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559552 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423836 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579445 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996748 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959257 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843681 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122687 s, 8.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135964 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142358 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16224 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175871 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193623 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170007 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207123 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199001 s, 5.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364835 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368887 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316924 s, 33.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369501 s, 28.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333582 s, 31.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367385 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418497 s, 25.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346149 s, 30.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366275 s, 28.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350633 s, 29.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365968 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355336 s, 29.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039261 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368894 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379454 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410152 s, 25.6 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg324-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg324-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9288 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9362 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg324-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg324-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg324-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg324-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg324-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.24144 s, 13.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567227 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112981 s, 9.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129765 s, 8.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125214 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102636 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128853 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138158 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145624 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155482 s, 6.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16678 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159135 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160069 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167293 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180685 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203806 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197084 s, 5.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028914 s, 36.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330893 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224478 s, 46.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336434 s, 31.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352338 s, 29.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307522 s, 34.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311559 s, 33.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329952 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430516 s, 24.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468043 s, 22.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478262 s, 21.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344828 s, 30.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477547 s, 22.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344187 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413387 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043532 s, 24.1 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg324-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg324-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8698 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9491 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (138s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 03:51:59 (1766393519) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg324-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.30661 s, 54.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058267 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616091 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566435 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072899 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095999 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654541 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773982 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11249 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103895 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142253 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120073 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146233 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127282 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126155 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144965 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14566 s, 7.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260083 s, 40.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195654 s, 53.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192189 s, 54.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178822 s, 58.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186881 s, 56.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366283 s, 28.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243803 s, 43.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02356 s, 44.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242333 s, 43.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270083 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305988 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272191 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030293 s, 34.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215185 s, 48.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320923 s, 32.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318864 s, 32.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg324-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg324-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.12244 s, 20.5 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.79519 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.24@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.24@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.183695 s, 91.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306083 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488359 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553532 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945868 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087393 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122039 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946612 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106909 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11559 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131654 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137703 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106537 s, 9.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147229 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148638 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128481 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111743 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00568233 s, 185 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205923 s, 50.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00990995 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311447 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00276954 s, 379 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384824 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249771 s, 42.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176279 s, 59.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00453381 s, 231 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318031 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00717985 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00446698 s, 235 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00495907 s, 211 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00440836 s, 238 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00365142 s, 287 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg324-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg324-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.11372 s, 20.5 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.83329 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.24@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg324-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.200015 s, 83.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305872 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506056 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621244 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192748 s, 54.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471483 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630482 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509863 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077609 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625073 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134871 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136945 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144554 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152104 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119042 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148411 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163972 s, 6.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024015 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234175 s, 44.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191249 s, 54.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218403 s, 48.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393192 s, 26.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122809 s, 85.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00639158 s, 164 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187379 s, 56.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00286492 s, 366 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00394356 s, 266 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00334284 s, 314 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00388287 s, 270 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00407629 s, 257 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00919937 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148234 s, 70.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00288462 s, 364 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg324-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg324-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.22123 s, 20.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.95726 s, 21.2 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg324-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.200895 s, 83.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368215 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346491 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045942 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101949 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126588 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111628 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122564 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137822 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109524 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110933 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102918 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124957 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142134 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135253 s, 7.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14161 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147145 s, 7.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330156 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339227 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134847 s, 77.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018065 s, 58.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153462 s, 68.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321451 s, 32.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00521463 s, 201 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425777 s, 24.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0068481 s, 153 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233481 s, 44.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505948 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239775 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00819989 s, 128 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00464743 s, 226 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00390368 s, 269 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00535128 s, 196 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg324-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg324-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90203 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.80764 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.08313 s, 20.6 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7849 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg324-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.856732 s, 19.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546156 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321725 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586545 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528613 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512119 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451787 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500961 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564098 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15188 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20885 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211865 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204871 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228074 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227967 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229963 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212547 s, 4.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171913 s, 61.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012223 s, 85.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169991 s, 61.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171042 s, 61.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203286 s, 51.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221477 s, 47.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121867 s, 86.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197748 s, 53.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225628 s, 46.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215024 s, 48.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019753 s, 53.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00452292 s, 232 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230854 s, 45.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234481 s, 44.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266766 s, 39.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023167 s, 45.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg324-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg324-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5486 s, 5.4 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4768 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg324-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.258452 s, 64.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316089 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558813 s, 18.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541498 s, 19.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534737 s, 19.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453592 s, 23.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678507 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690751 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529307 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799044 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848864 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952822 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121764 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121373 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104741 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978361 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13519 s, 7.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117691 s, 89.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291397 s, 36.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141953 s, 73.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146756 s, 71.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233076 s, 45.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224206 s, 46.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251949 s, 41.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163433 s, 64.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300425 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248209 s, 42.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397754 s, 26.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00481905 s, 218 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00759194 s, 138 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00550418 s, 191 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0059255 s, 177 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101574 s, 103 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg324-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg324-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.15373 s, 20.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.75974 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg324-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.175331 s, 95.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035617 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379333 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453853 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414792 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380538 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246063 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440563 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650077 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737879 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868705 s, 12.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988207 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991761 s, 10.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934015 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925829 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0997902 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923779 s, 11.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138146 s, 75.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208495 s, 50.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00380072 s, 276 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00379314 s, 276 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00274794 s, 382 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00475772 s, 220 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00570873 s, 184 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389651 s, 26.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026994 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00571933 s, 183 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235848 s, 44.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435652 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274555 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221188 s, 47.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00941386 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00245563 s, 427 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg324-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg324-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.8926 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.78933 s, 10.7 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.86908 s, 10.6 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.8124 s, 10.7 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (365s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 03:58:04 (1766393884) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg324-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.269707 s, 62.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330372 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817839 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851639 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825938 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116326 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112709 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127653 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117 s, 9.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141474 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131924 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137486 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128098 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151581 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154668 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151065 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168542 s, 6.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297791 s, 35.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415919 s, 25.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267766 s, 39.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323096 s, 32.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238234 s, 44.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267892 s, 39.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285571 s, 36.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283009 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341357 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298576 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302491 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037872 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374838 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325079 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329585 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381019 s, 27.5 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg324-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg324-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.18792 s, 20.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.78275 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (40s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 03:58:45 (1766393925) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.24@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.24@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.344938 s, 48.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729105 s, 14.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058195 s, 18.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402823 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717201 s, 14.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800142 s, 13.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074183 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809076 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113766 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935314 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120717 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115693 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667959 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213105 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193797 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219981 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274644 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176065 s, 59.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136806 s, 76.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238705 s, 43.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300979 s, 34.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132004 s, 79.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00961813 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207004 s, 50.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00482004 s, 218 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297703 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00388734 s, 270 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334936 s, 31.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00469285 s, 223 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263431 s, 39.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00423168 s, 248 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00993653 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291178 s, 36.0 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg324-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg324-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.3752 s, 19.5 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9214 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (45s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 03:59:30 (1766393970) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.24@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg324-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.469306 s, 35.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371453 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566281 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541493 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115171 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156785 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139857 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163712 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16223 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162213 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133957 s, 7.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17394 s, 6.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164759 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169243 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125959 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175002 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167701 s, 6.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103626 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00350896 s, 299 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00398859 s, 263 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922611 s, 11.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740845 s, 14.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628435 s, 16.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503018 s, 20.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605862 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293894 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684474 s, 15.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122225 s, 85.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346669 s, 30.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319382 s, 32.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00924565 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229636 s, 45.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109119 s, 96.1 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg324-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg324-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.88188 s, 17.8 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.03943 s, 20.8 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (38s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 04:00:08 (1766394008) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg324-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.557098 s, 30.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485656 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450061 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633178 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468329 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799305 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925175 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103552 s, 10.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134616 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113333 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133325 s, 7.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102486 s, 10.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15654 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155873 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121835 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274389 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262599 s, 4.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245979 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00708289 s, 148 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00494829 s, 212 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00886447 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00495716 s, 212 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00515993 s, 203 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00442366 s, 237 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441905 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452918 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452807 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00325087 s, 323 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373678 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344737 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00670518 s, 156 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508811 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621708 s, 16.9 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg324-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg324-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99095 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.82555 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.38336 s, 19.5 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.85666 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (82s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 04:01:30 (1766394090) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg324-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.980408 s, 17.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486149 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659844 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119207 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168361 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163065 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158632 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15012 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163425 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177005 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17122 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176021 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194464 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184167 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191773 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176896 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191085 s, 5.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034407 s, 30.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286324 s, 36.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289017 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661665 s, 15.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425885 s, 24.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042691 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730989 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779583 s, 13.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386644 s, 27.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369538 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395878 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426923 s, 24.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442674 s, 23.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583657 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787898 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559979 s, 18.7 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg324-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg324-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6508 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9547 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (66s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 04:02:36 (1766394156) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg324-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.300759 s, 55.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197653 s, 53.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022187 s, 47.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265471 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379014 s, 27.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12664 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115296 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137779 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135322 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12636 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118785 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13088 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145315 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12279 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118686 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127521 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15393 s, 6.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00667941 s, 157 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177676 s, 59.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310791 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237587 s, 44.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290398 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271031 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191206 s, 54.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00556282 s, 188 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00815877 s, 129 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402567 s, 26.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282601 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00543645 s, 193 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428518 s, 24.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530394 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00637886 s, 164 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00719665 s, 146 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg324-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg324-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.14274 s, 20.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7765 s, 10.7 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg324-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.171423 s, 97.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270938 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361632 s, 29.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496037 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453066 s, 23.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453907 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740253 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721488 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959359 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887151 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889375 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807955 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173349 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191934 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178437 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180977 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171661 s, 6.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111703 s, 93.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175214 s, 59.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166377 s, 63.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302103 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129128 s, 81.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140564 s, 74.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199457 s, 52.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205298 s, 51.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214176 s, 49.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219892 s, 47.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230266 s, 45.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221077 s, 47.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028768 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255207 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258347 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319412 s, 32.8 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg324-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg324-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96669 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.81511 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.86847 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.79408 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (122s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 04:04:38 (1766394278) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (4s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 04:04:42 (1766394282) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg324-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.45291 s, 304 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg324-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0674 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (52s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 04:05:34 (1766394334) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.167891 s, 99.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301219 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290661 s, 36.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640124 s, 16.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648854 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107949 s, 9.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8cd00388b800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff8cd00388b800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106479 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969391 s, 10.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113889 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123798 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10792 s, 9.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121775 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101785 s, 10.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125707 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1115 s, 9.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122644 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121757 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011294 s, 92.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162933 s, 64.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8cd00388b800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445746 s, 23.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421816 s, 24.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361117 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00382202 s, 274 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00291033 s, 360 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054184 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262611 s, 39.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020038 s, 52.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217906 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184508 s, 56.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180164 s, 58.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173378 s, 60.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113493 s, 92.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129959 s, 80.7 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg324-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg324-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.23687 s, 20.0 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.00875 s, 20.9 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.160331 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357944 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035953 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948672 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859588 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937956 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995716 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963731 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113149 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100545 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120223 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1264 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115578 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111415 s, 9.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122039 s, 8.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127016 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107676 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020735 s, 50.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119702 s, 87.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287363 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02015 s, 52.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025195 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196055 s, 53.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197088 s, 53.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212355 s, 49.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263183 s, 39.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306187 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267383 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237394 s, 44.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299169 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252392 s, 41.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280498 s, 37.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281747 s, 37.2 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg324-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg324-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.04833 s, 20.8 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.99921 s, 21.0 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (64s) == sanityn test 77o: Changing rank should not panic ====== 04:06:39 (1766394399) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (5s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 04:06:43 (1766394403) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (42s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 04:07:25 (1766394445) mds.MDS.mdt.nrs_policies=tbf oleg324-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg324-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg324-client: oleg324-server: ssh exited with exit code 36 oleg324-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg324-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg324-client: oleg324-server: ssh exited with exit code 22 oleg324-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg324-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg324-client: oleg324-server: ssh exited with exit code 22 oleg324-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg324-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg324-client: oleg324-server: ssh exited with exit code 22 oleg324-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg324-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg324-client: oleg324-server: ssh exited with exit code 22 oleg324-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg324-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg324-client: oleg324-server: ssh exited with exit code 22 oleg324-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg324-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg324-client: oleg324-server: ssh exited with exit code 22 oleg324-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg324-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg324-client: oleg324-server: ssh exited with exit code 22 oleg324-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg324-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg324-client: oleg324-server: ssh exited with exit code 22 oleg324-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg324-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg324-client: oleg324-server: ssh exited with exit code 22 oleg324-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg324-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg324-client: oleg324-server: ssh exited with exit code 22 oleg324-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg324-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg324-client: oleg324-server: ssh exited with exit code 22 oleg324-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg324-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg324-client: oleg324-server: ssh exited with exit code 22 oleg324-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg324-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg324-client: oleg324-server: ssh exited with exit code 22 oleg324-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg324-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg324-client: oleg324-server: ssh exited with exit code 22 oleg324-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg324-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg324-client: oleg324-server: ssh exited with exit code 22 oleg324-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg324-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg324-client: oleg324-server: ssh exited with exit code 22 oleg324-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg324-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg324-client: oleg324-server: ssh exited with exit code 22 oleg324-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg324-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg324-client: oleg324-server: ssh exited with exit code 22 oleg324-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg324-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg324-client: oleg324-server: ssh exited with exit code 22 oleg324-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg324-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg324-client: oleg324-server: ssh exited with exit code 22 oleg324-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg324-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg324-client: oleg324-server: ssh exited with exit code 22 oleg324-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg324-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg324-client: oleg324-server: ssh exited with exit code 22 oleg324-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg324-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg324-client: oleg324-server: ssh exited with exit code 22 oleg324-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg324-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg324-client: oleg324-server: ssh exited with exit code 22 oleg324-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg324-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg324-client: oleg324-server: ssh exited with exit code 22 oleg324-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg324-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg324-client: oleg324-server: ssh exited with exit code 22 oleg324-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg324-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg324-client: oleg324-server: ssh exited with exit code 22 oleg324-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg324-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg324-client: oleg324-server: ssh exited with exit code 22 oleg324-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg324-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg324-client: oleg324-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (13s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 04:07:38 (1766394458) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 308798 kill md_thread xargs PASS 77r (42s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 04:08:20 (1766394500) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (3s) == sanityn test 79: xattr: intent error ================== 04:08:23 (1766394503) File: /mnt/lustre/d79.sanityn Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272506413 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 04:08:24.000000000 -0500 Modify: 2025-12-22 04:08:24.000000000 -0500 Change: 2025-12-22 04:08:24.000000000 -0500 Birth: 2025-12-22 04:08:24.000000000 -0500 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (13s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 04:08:36 (1766394516) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (1s) == sanityn test 80b: Accessing directory during migration ========================================================== 04:08:37 (1766394517) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (2s) == sanityn test 81a: rename and stat under striped directory ========================================================== 04:08:39 (1766394519) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (1s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 04:08:40 (1766394520) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (1s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 04:08:41 (1766394521) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (1s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 04:08:42 (1766394522) - open/close 2418 (time 1766394534.41 total 10.00 last 241.72) total: 2500 open/close in 10.38 seconds: 240.78 ops/second snapshot_time 1766394556.693661522 secs.nsecs start_time 1766394523.982708556 secs.nsecs elapsed_time 32.710952966 secs.nsecs open 2505 samples [usecs] 122 5430 1508547 1042105417 close 2504 samples [usecs] 85 1301 498152 116758154 mknod 2501 samples [usecs] 209 3797 1314809 790743975 unlink 2500 samples [usecs] 297 35365 1887259 4848911843 mkdir 2 samples [usecs] 763 1054 1817 1693085 rmdir 2 samples [usecs] 2507 3641 6148 19541930 rename 2500 samples [usecs] 308 8352 4004398 9278796958 getattr 3762 samples [usecs] 6 1166 104593 5236443 parallel_rename_file 2500 samples [usecs] 308 8352 4004398 9278796958 crossdir_rename 2500 samples [usecs] 308 8352 4004398 9278796958 PASS 81d (36s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 04:09:18 (1766394558) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7552 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7552 PASS 82 (2s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 04:09:20 (1766394560) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (1s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 04:09:22 (1766394562) fail_loc=0x8000060b PASS 84 (8s) == sanityn test 85: Lustre API root cache race =========== 04:09:29 (1766394569) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1766394570 Finishing test test1 at 1766394570 PASS 85 (3s) == sanityn test 90: open/create and unlink striped directory ========================================================== 04:09:32 (1766394572) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (1s) == sanityn test 91: chmod and unlink striped directory === 04:09:33 (1766394573) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (2s) == sanityn test 92: create remote directory under orphan directory ========================================================== 04:09:35 (1766394575) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (1s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 04:09:36 (1766394576) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5718 0x1656 0x280000400 0 5713 0x1651 0x240000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5719 0x1657 0x280000400 0 5714 0x1652 0x240000400 rc1=0 and rc2=0 PASS 93 (9s) == sanityn test 94: signal vs CP callback race =========== 04:09:45 (1766394585) osc.lustre-OST0000-osc-ffff8cd00388b800.idle_timeout=0 osc.lustre-OST0000-osc-ffff8cd018895000.idle_timeout=0 osc.lustre-OST0001-osc-ffff8cd00388b800.idle_timeout=0 osc.lustre-OST0001-osc-ffff8cd018895000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0682203 s, 120 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.09959 s, 1.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5988: 326856 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0138106 s, 297 kB/s osc.lustre-OST0000-osc-ffff8cd00388b800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8cd018895000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8cd00388b800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8cd018895000.idle_timeout=debug PASS 94 (10s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 04:09:55 (1766394595) llite.lustre-ffff8cd00388b800.fast_read=0 llite.lustre-ffff8cd018895000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0200715 s, 104 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff8cd00388b800.fast_read=1 llite.lustre-ffff8cd018895000.fast_read=1 PASS 95a (15s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 04:10:10 (1766394610) llite.lustre-ffff8cd00388b800.fast_read=0 llite.lustre-ffff8cd018895000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00200639 s, 6.1 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff8cd00388b800.fast_read=1 llite.lustre-ffff8cd018895000.fast_read=1 PASS 95b (8s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 04:10:18 (1766394618) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (1s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 04:10:19 (1766394619) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00173241 s, 2.4 MB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (3s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 04:10:22 (1766394622) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0482405 s, 43.5 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (2s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 04:10:24 (1766394624) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0575636 s, 36.4 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (2s) == sanityn test 100e: DoM: read on open and file size ==== 04:10:26 (1766394626) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00176657 s, 6.8 MB/s PASS 100e (3s) == sanityn test 101a: Discard DoM data on unlink ========= 04:10:29 (1766394629) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000531232 s, 7.7 MB/s PASS 101a (2s) == sanityn test 101b: Discard DoM data on rename ========= 04:10:31 (1766394631) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000577193 s, 7.1 MB/s PASS 101b (3s) == sanityn test 101c: Discard DoM data on close-unlink === 04:10:34 (1766394634) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000727539 s, 5.6 MB/s PASS 101c (4s) SKIP: sanityn test_102 skipping ALWAYS excluded test 102 == sanityn test 103: Test size correctness with lockahead ========================================================== 04:10:38 (1766394638) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1766394639 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1766394642 Finishing test test23 at 1766394642 PASS 103 (8s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 04:10:46 (1766394646) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (1s) == sanityn test 105: Glimpse and lock cancel race ======== 04:10:47 (1766394647) fail_loc=0x416 PASS 105 (23s) == sanityn test 106a: Verify the btime via statx() ======= 04:11:10 (1766394670) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (1s) == sanityn test 106b: Glimpse RPCs test for statx ======== 04:11:11 (1766394671) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00533506 s, 192 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205272508948 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 04:11:12.000000000 -0500 Modify: 2025-12-22 04:11:12.000000000 -0500 Change: 2025-12-22 04:11:12.000000000 -0500 Birth: 2025-12-22 04:11:12.000000000 -0500 /mnt/lustre/f106b.sanityn 144115205272508948 -rw-r--r-- 1 0 0 1766394672 1766394672 1766394672 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205272508948 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 04:11:12.000000000 -0500 Modify: 2025-12-22 04:11:12.000000000 -0500 Change: 2025-12-22 04:11:12.000000000 -0500 Birth: 2025-12-22 04:11:12.000000000 -0500 1766394672 1024 1 PASS 106b (3s) == sanityn test 106c: Verify statx attributes mask ======= 04:11:14 (1766394674) PASS 106c (3s) == sanityn test 107a: Basic grouplock conflict =========== 04:11:17 (1766394677) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.108888 s, 96.3 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5722 0x165a 0x240000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7552 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7552 First grouplock blocks second one PASS 107a (4s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 04:11:22 (1766394682) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.107195 s, 97.8 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5726 0x165e 0x280000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7552 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (9s) == sanityn test 108a: lseek: parallel updates ============ 04:11:30 (1766394690) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00251485 s, 4.0 MB/s llite.lustre-ffff8cd00388b800.hybrid_io=0 llite.lustre-ffff8cd018895000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.36733 s, 1.9 MB/s PASS 108a (7s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 04:11:37 (1766394697) 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre (opts:) 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) Starting race between client mount instances (50 iterations): Iteration 1 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre (opts:) Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 2 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre (opts:) Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 3 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre (opts:) Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 4 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre (opts:) Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 5 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg324-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 6 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg324-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 7 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre (opts:) Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 8 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre (opts:) Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 9 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre (opts:) Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 10 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre (opts:) Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 11 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg324-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 12 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg324-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 13 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre (opts:) Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 14 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre (opts:) Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 15 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg324-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 16 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre (opts:) Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 17 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg324-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 18 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre (opts:) Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 19 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre (opts:) Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 20 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg324-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 21 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre (opts:) Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 22 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre (opts:) Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 23 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre (opts:) Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 24 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre (opts:) Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 25 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre (opts:) Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 26 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre (opts:) Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 27 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg324-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 28 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg324-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 29 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre (opts:) Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 30 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg324-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 31 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg324-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 32 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg324-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 33 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre (opts:) Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 34 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg324-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 35 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre (opts:) Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 36 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg324-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 37 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre (opts:) Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 38 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg324-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 39 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre (opts:) Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 40 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg324-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 41 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg324-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 42 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre (opts:) Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 43 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre (opts:) Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 44 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre (opts:) Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 45 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg324-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 46 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre (opts:) Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 47 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg324-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 48 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre (opts:) Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 49 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre (opts:) Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Iteration 50 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 192.168.203.124@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.124@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg324-client.virtnet /mnt/lustre (opts:) Stopping client oleg324-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 PASS 109 (515s) == sanityn test 110: do not grant another lock on resend ========================================================== 04:20:12 (1766395212) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205306056707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-22 04:20:13.000000000 -0500 Modify: 2025-12-22 04:20:13.000000000 -0500 Change: 2025-12-22 04:20:13.000000000 -0500 Birth: 2025-12-22 04:20:13.000000000 -0500 fail_loc=0 fail_val=0 Connected clients: oleg324-client.virtnet oleg324-client.virtnet PASS 110 (65s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 04:21:18 (1766395278) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (1s) == sanityn test 112: update max-inherit in default LMV === 04:21:19 (1766395279) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (1s) == sanityn test 113: check servers of specified fs ======= 04:21:20 (1766395280) lustre-OST0000-osc-ffff8cd00388f000 active. lustre-OST0001-osc-ffff8cd00388f000 active. lustre-OST0000-osc-ffff8cd00388f000 active. lustre-OST0001-osc-ffff8cd00388f000 active. MGC192.168.203.124@tcp active. PASS 113 (2s) == sanityn test 114: implicit default LMV inherit ======== 04:21:22 (1766395282) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (1s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 04:21:23 (1766395283) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (1s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 04:21:24 (1766395284) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (1s) == sanityn test 117a: TCU: Init and enable Trash Can on MDTs ========================================================== 04:21:25 (1766395285) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_enable=0 PASS 117a (3s) == sanityn test 117b: Move regular file and empty dir into trash can dir ========================================================== 04:21:28 (1766395288) SKIP: sanityn test_117b ldiskfs only test SKIP 117b (1s) == sanityn test 117c: Move deleted tree with multiple levels into trash ========================================================== 04:21:29 (1766395289) SKIP: sanityn test_117c ldiskfs only test SKIP 117c (1s) == sanityn test 117d: Per-User Trash can Type testing ==== 04:21:31 (1766395291) SKIP: sanityn test_117d ldiskfs only test SKIP 117d (1s) == sanityn test 117e: Undeleted dir in trash should keep its original xattrs ========================================================== 04:21:32 (1766395292) SKIP: sanityn test_117e ldiskfs only test SKIP 117e (1s) == sanityn test 117f: Uncache the dentry under the trash dir ========================================================== 04:21:33 (1766395293) SKIP: sanityn test_117f ldiskfs only test SKIP 117f (1s) == sanityn test 117g: Access .Trash for a non-striped directory ========================================================== 04:21:34 (1766395294) SKIP: sanityn test_117g ldiskfs only test SKIP 117g (1s) == sanityn test 117h: Permission testing of various operations for files in Trash Can ========================================================== 04:21:35 (1766395295) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain Trash Can Type: plain Trash list: ===> /mnt/lustre/.lustre/trash/MDT0000: 0x200000403:0x5:0x0 /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0: tdir1 tfile3 /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0/tdir1: <=== touch: setting times of '/mnt/lustre/.lustre/trash/f117h.sanityn.crfail': No such file or directory mkdir: cannot create directory '/mnt/lustre/.lustre/trash/f117h.sanityn.crfail': Operation not permitted touch: setting times of '/mnt/lustre/.lustre/trash/MDT0000/f117h.sanityn.crfail': No such file or directory mkdir: cannot create directory '/mnt/lustre/.lustre/trash/MDT0000/f117h.sanityn.crfail': Operation not permitted rmdir: failed to remove '/mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0': Operation not permitted touch: setting times of '/mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0/f117h.sanityn.crfail': No such file or directory mv: cannot move '/mnt/lustre/d117h.sanityn/f117h.sanityn.mvfail' to '/mnt/lustre/.lustre/trash/f117h.sanityn.mvfail': Operation not permitted mv: cannot move '/mnt/lustre/d117h.sanityn/f117h.sanityn.mvfail' to '/mnt/lustre/.lustre/trash/MDT0000/f117h.sanityn.mvfail': No data available mv: cannot move '/mnt/lustre/d117h.sanityn/f117h.sanityn.mvfail' to '/mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0/f117h.sanityn.mvfail': No data available mv: cannot move '/mnt/lustre/d117h.sanityn/d117h.sanityn.mvfail' to '/mnt/lustre/.lustre/trash/d117h.sanityn.mvfail': Operation not permitted mv: cannot move '/mnt/lustre/d117h.sanityn/d117h.sanityn.mvfail' to '/mnt/lustre/.lustre/trash/MDT0000/d117h.sanityn.mvfail': No data available mv: cannot move '/mnt/lustre/d117h.sanityn/d117h.sanityn.mvfail' to '/mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0/d117h.sanityn.mvfail': No data available chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tfile3 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tfile3 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tfile3 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tfile3 chmod: changing permissions of '/mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tfile3': Bad file descriptor truncate: cannot truncate '/mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tfile3' to length 4096: Bad file descriptor chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tdir1 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tdir1 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tdir1 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tdir1 chmod: changing permissions of '/mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tdir1': Bad file descriptor setfattr: /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tfile3: Bad file descriptor setfattr: /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tfile3: Bad file descriptor setfattr: /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tdir1: Bad file descriptor setfattr: /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tdir1: Bad file descriptor open: Bad file descriptor open: Bad file descriptor read: Operation not permitted mmap: Operation not permitted open: Is a directory open: Is a directory mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid PASS 117h (5s) == sanityn test 117i: Forbid to create a file with name .Trash ========================================================== 04:21:40 (1766395300) mkdir: cannot create directory '/mnt/lustre/d117i.sanityn/.Trash': Invalid argument touch: setting times of '/mnt/lustre/d117i.sanityn/.Trash': No such file or directory open(O_RDWR|O_CREAT): Invalid argument mcreate: cannot create `/mnt/lustre/d117i.sanityn/.Trash' with mode 0100644: Invalid argument ln: failed to create hard link '/mnt/lustre/d117i.sanityn/.Trash' => '/mnt/lustre/d117i.sanityn/f117i.sanityn': Invalid argument ln: failed to create symbolic link '/mnt/lustre/d117i.sanityn/.Trash': Invalid argument mv: cannot move '/mnt/lustre/d117i.sanityn/f117i.sanityn' to a subdirectory of itself, '/mnt/lustre/d117i.sanityn/.Trash' mv: cannot move '/mnt/lustre/d117i.sanityn/f117i.sanityn' to a subdirectory of itself, '/mnt/lustre/.Trash' ln: failed to create symbolic link '/mnt/lustre/d117i.sanityn/.Trash': Invalid argument mv: cannot move '/mnt/lustre/d117i.sanityn/d117i.sanityn' to a subdirectory of itself, '/mnt/lustre/d117i.sanityn/.Trash' mv: cannot move '/mnt/lustre/d117i.sanityn/d117i.sanityn' to a subdirectory of itself, '/mnt/lustre/.Trash' PASS 117i (2s) == sanityn test 117j: Move back file from Trash Can ====== 04:21:42 (1766395302) SKIP: sanityn test_117j ldiskfs only test SKIP 117j (1s) == sanityn test 117k: Unrm file from Trash Can =========== 04:21:43 (1766395303) SKIP: sanityn test_117k ldiskfs only test SKIP 117k (1s) == sanityn test 117l: Clean up files in Trash Can via command 'lfs trash clean' ========================================================== 04:21:44 (1766395304) SKIP: sanityn test_117l ldiskfs only test SKIP 117l (1s) == sanityn test 117m: Empty Trash Can via the command 'lfs trash clean' ========================================================== 04:21:46 (1766395306) SKIP: sanityn test_117m ldiskfs only test SKIP 117m (0s) == sanityn test 117n: Update LinkEA accordingly when moving file into Trash Can ========================================================== 04:21:47 (1766395307) SKIP: sanityn test_117n ldiskfs only test SKIP 117n (1s) == sanityn test 117o: Set/Clear NOTRASH flag on file ===== 04:21:48 (1766395308) SKIP: sanityn test_117o ldiskfs only test SKIP 117o (1s) == sanityn test 117p: Trash Can handling for remote dir/file ========================================================== 04:21:49 (1766395309) SKIP: sanityn test_117p needs >= 2 MDTs SKIP 117p (1s) == sanityn test 117pb: Trash Can handling for remote dir/file with multiple levels ========================================================== 04:21:50 (1766395310) SKIP: sanityn test_117pb needs >= 2 MDTs SKIP 117pb (1s) == sanityn test 117q: Access Trash Can via '.Trash' for striped directory ========================================================== 04:21:51 (1766395311) SKIP: sanityn test_117q needs >= 2 MDTs SKIP 117q (1s) == sanityn test 117r: Verify the deleting a striped directory on Trash Can ========================================================== 04:21:52 (1766395312) SKIP: sanityn test_117r ldiskfs only test SKIP 117r (1s) == sanityn test 117s: Create trash shard if it does not exist ========================================================== 04:21:53 (1766395313) SKIP: sanityn test_117s ldiskfs only test SKIP 117s (1s) == sanityn test 117t: UID/GID/ProjID for files in Trash Can ========================================================== 04:21:54 (1766395314) SKIP: sanityn test_117t ldiskfs only test SKIP 117t (1s) == sanityn test 117u: statfs() call should not account the space in Trash Can ========================================================== 04:21:55 (1766395315) SKIP: sanityn test_117u ldiskfs only test SKIP 117u (2s) == sanityn test 117ub: statfs() account the inode usage in Trash Can ========================================================== 04:21:57 (1766395317) SKIP: sanityn test_117ub ldiskfs only test SKIP 117ub (1s) == sanityn test 117v: Test for Trash Can UID/GID/PROJID setting ========================================================== 04:21:58 (1766395318) SKIP: sanityn test_117v ldiskfs only test SKIP 117v (2s) == sanityn test 117w: Test Trash Can with original UID/GID/PROJID ========================================================== 04:22:00 (1766395320) SKIP: sanityn test_117w ldiskfs only test SKIP 117w (1s) == sanityn test 117x: Unrm Files from Trash Can should also restore UID/GID/PROJID ========================================================== 04:22:01 (1766395321) SKIP: sanityn test_117x ldiskfs only test SKIP 117x (1s) == sanityn test 117ya: Test with symbol links with Trash Can ========================================================== 04:22:02 (1766395322) SKIP: sanityn test_117ya ldiskfs only test SKIP 117ya (1s) == sanityn test 117yb: Compare the directories containing symlinks after unrm recover ========================================================== 04:22:03 (1766395323) SKIP: sanityn test_117yb ldiskfs only test SKIP 117yb (1s) == sanityn test 117yc: Restoring UID/GID/PROJID for symlinks works ========================================================== 04:22:04 (1766395324) SKIP: sanityn test_117yc ldiskfs only test SKIP 117yc (2s) == sanityn test 117za: Test Trash Can for UID Trash Can type with nodemap ========================================================== 04:22:06 (1766395326) SKIP: sanityn test_117za ldiskfs only test SKIP 117za (1s) == sanityn test 117zb: Test Trash Can for PLAIN Trash Can type with nodemap ========================================================== 04:22:07 (1766395327) SKIP: sanityn test_117zb ldiskfs only test SKIP 117zb (1s) == sanityn test 117zc: TCU: Test striped dir for UID Trash Can type ========================================================== 04:22:08 (1766395328) SKIP: sanityn test_117zc ldiskfs only test SKIP 117zc (1s) == sanityn test 117zd: TCU: Test striped dir for UID Trash Can type with nodemap ========================================================== 04:22:09 (1766395329) On MGS 192.168.203.124, active = nodemap.active=1 waiting 10 secs for sync On MGS 192.168.203.124, tnm.trusted_nodemap = nodemap.tnm.trusted_nodemap=1 waiting 10 secs for sync nodemap.tnm.idmap= [ { idtype: uid, client_id: 500, fs_id: 10000 }, { idtype: uid, client_id: 501, fs_id: 10001 }, { idtype: uid, client_id: 502, fs_id: 10002 }, { idtype: uid, client_id: 503, fs_id: 10003 }, { idtype: uid, client_id: 504, fs_id: 10004 }, { idtype: uid, client_id: 505, fs_id: 10005 }, { idtype: uid, client_id: 506, fs_id: 10006 }, { idtype: uid, client_id: 507, fs_id: 10007 }, { idtype: uid, client_id: 508, fs_id: 10008 }, { idtype: uid, client_id: 509, fs_id: 10009 } ] SKIP: sanityn test_117zd ldiskfs only test On MGS 192.168.203.124, active = nodemap.active=0 waiting 10 secs for sync SKIP 117zd (38s) == sanityn test 118Aa: TCU: Orphan object handling for a regular file or an empty dir ========================================================== 04:22:47 (1766395367) SKIP: sanityn test_118Aa ldiskfs only test SKIP 118Aa (1s) == sanityn test 118Ab: TCU: Orphan object handling for non-emtpy dir ========================================================== 04:22:48 (1766395368) SKIP: sanityn test_118Ab ldiskfs only test SKIP 118Ab (1s) == sanityn test 118Ba: TCU: 'oldest' repeated deletion for a regular file or empty dir ========================================================== 04:22:49 (1766395369) SKIP: sanityn test_118Ba ldiskfs only test SKIP 118Ba (2s) == sanityn test 118Bb: TCU: 'timestamp' repeated deletion policy ========================================================== 04:22:51 (1766395371) SKIP: sanityn test_118Bb ldiskfs only test SKIP 118Bb (1s) == sanityn test 118C: TCU: add support for rename() ====== 04:22:52 (1766395372) SKIP: sanityn test_118C ldiskfs only test SKIP 118C (2s) == sanityn test 118d: TCU: repeated deletion of same fname for a empty dir in DNE env ========================================================== 04:22:54 (1766395374) SKIP: sanityn test_118d ldiskfs only test SKIP 118d (1s) == sanityn test 118e: TCU: Repeated deletion of same name for a non-empty remote dir ========================================================== 04:22:55 (1766395375) SKIP: sanityn test_118e ldiskfs only test SKIP 118e (2s) == sanityn test 118f: Avoid caching negative dentry cache for dir in Trash Can ========================================================== 04:22:57 (1766395377) SKIP: sanityn test_118f ldiskfs only test SKIP 118f (1s) == sanityn test 200: service remains healthy while able to process request ========================================================== 04:22:58 (1766395378) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=20 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0933 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0954 s, 0.2 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0875 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0978 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0937 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1043 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0917 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0968 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1044 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1603 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1205 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1419 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.17 s, 0.1 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1737 s, 0.1 kB/s1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1342 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1328 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1671 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1421 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2352 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2525 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2125 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2142 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.266 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2391 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2466 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2217 s, 0.1 kB/s1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2232 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.2926 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3461 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.2916 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3249 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3038 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3286 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3226 s, 0.1 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3286 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3788 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 100.389 s, 0.0 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 100.403 s, 0.0 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 100.412 s, 0.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 100.444 s, 0.0 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg324-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8cd00388f000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8cd00388f000.ost_server_uuid in FULL state after 0 sec PASS 200 (107s) cleanup: ====================================================== == sanityn test complete, duration 6979 sec ============== 04:24:45 (1766395485) === sanityn: start cleanup 04:24:46 (1766395486) === Stopping clients: oleg324-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg324-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 04:25:31 (1766395531) ===