-----============= acceptance-small: sanityn ============----- Sun May 17 21:40:25 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg230-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a 102 skipping tests SLOW=no: 33a === sanityn: start setup 21:40:49 (1779068449) === Starting client oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 Mount client oleg230-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 Started clients oleg230-client.virtnet: 192.168.202.130@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg230-client.virtnet: executing check_config_client /mnt/lustre oleg230-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg230-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff923242ae4000.idle_timeout=debug osc.lustre-OST0000-osc-ffff92325195d800.idle_timeout=debug osc.lustre-OST0001-osc-ffff923242ae4000.idle_timeout=debug osc.lustre-OST0001-osc-ffff92325195d800.idle_timeout=debug disable quota as required oleg230-server: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 21:41:31 (1779068491) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312101 s, 33.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7571] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 21:41:35 (1779068495) PASS 0 (9s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 21:41:44 (1779068504) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (9s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 21:41:53 (1779068513) -rw-r--r-- 1 root root 0 May 17 21:41 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (8s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 21:42:01 (1779068521) -rw-r--r-- 1 root root 0 May 17 21:42 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (7s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 21:42:08 (1779068528) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (7s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 21:42:15 (1779068535) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (8s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 21:42:23 (1779068543) total 1 -rwxrwxrwx 1 root root 0 May 17 21:41 f2a -rwxrwxrwx 1 root root 0 May 17 21:42 f2b total 1 -rwxrwxrwx 1 root root 0 May 17 21:41 f2a -rwxrwxrwx 1 root root 0 May 17 21:42 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0117879 s, 43.4 kB/s PASS 2e (9s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 21:42:32 (1779068552) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (4s) == sanityn test 2g: check blocks update on sync write ==== 21:42:37 (1779068557) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.68601 s, 3.1 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (9s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 21:42:46 (1779068566) PASS 3 (9s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 21:42:55 (1779068575) PASS 4 (9s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 21:43:05 (1779068585) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (9s) == sanityn test 6: remove of open file on other node ============================================================================ 21:43:15 (1779068595) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (9s) == sanityn test 7: remove of open directory on other node ======================================================================= 21:43:23 (1779068603) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (9s) == sanityn test 8: remove of open special file on other node ==================================================================== 21:43:32 (1779068612) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (10s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 21:43:42 (1779068622) PASS 9a (10s) == sanityn test 9b: append to striped sparse file ======== 21:43:53 (1779068633) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101269 s, 10.4 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0593095 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (14s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 21:44:06 (1779068646) 1+0 records in 1+0 records out 1 byte copied, 0.00764073 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0170159 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0160551 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0201926 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0143666 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0228566 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0223729 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0129795 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0185788 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0117857 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0102711 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0185405 s, 0.1 kB/s PASS 10a (11s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 21:44:17 (1779068657) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0634535 s, 48.4 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.044783 s, 91.5 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00288527 s, 1.1 MB/s PASS 10b (8s) == sanityn test 11: execution of file opened for write should return error ============================================================== 21:44:25 (1779068665) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7571 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 300: /mnt/lustre2/d11/f: Text file busy PASS 11 (8s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 21:44:33 (1779068673) warning: '-runtime' deprecated, use '-t runtime' instead - link 342 (time 1779068686.08 total 10.00 last 34.19) - link 593 (time 1779068696.08 total 20.01 last 25.09) - link 893 (time 1779068706.09 total 30.01 last 29.98) - link 1183 (time 1779068716.10 total 40.03 last 28.96) - link 1455 (time 1779068726.10 total 50.03 last 27.19) - link 1759 (time 1779068736.16 total 60.09 last 30.21) - link 2118 (time 1779068746.19 total 70.12 last 35.80) - link 2454 (time 1779068756.20 total 80.12 last 33.59) - link 2767 (time 1779068766.21 total 90.13 last 31.27) total: 3113 link in 100.02 seconds: 31.12 ops/second using seed 184662860 running for 100 seconds - stat 5761 (time 1779068688 ; total 11 ; last 11) - stat 10000 (time 1779068696 ; total 19 ; last 8) - stat 16054 (time 1779068707 ; total 30 ; last 11) - stat 20000 (time 1779068714 ; total 37 ; last 7) - stat 25870 (time 1779068725 ; total 48 ; last 11) - stat 29441 (time 1779068736 ; total 59 ; last 11) - stat 30000 (time 1779068737 ; total 60 ; last 1) - stat 33642 (time 1779068748 ; total 71 ; last 11) - stat 37003 (time 1779068759 ; total 82 ; last 11) - stat 40000 (time 1779068767 ; total 90 ; last 8) total: 43631 stats in 100 seconds: 436.309998 stats/second - unlinked 0 (time 1779068727 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3113) error: No such file or directory total: 3112 unlinks in 78 seconds: 39.897434 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21381) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21383) - No such process PASS 12 (136s) == sanityn test 13: test directory page revocation ======= 21:46:50 (1779068810) f13 f13 PASS 13 (8s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 21:46:57 (1779068817) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7571 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 335: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 22567 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (6s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 21:47:03 (1779068823) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 23147 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (6s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 21:47:09 (1779068829) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 350: 23725 Terminated $DIR1/$tdir/sleep 60 PASS 14b (6s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 21:47:16 (1779068836) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 362: 24307 Terminated $DIR1/$tdir/sleep 60 PASS 14c (7s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 21:47:23 (1779068843) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 373: 24890 Terminated $DIR1/$tdir/sleep 60 PASS 14d (7s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 21:47:30 (1779068850) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7528448 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7528448KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (48s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 21:48:18 (1779068898) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.655643 s, 12.8 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.437624 s, 19.2 MB/s Chance of close/open is 1 in 50 Seed set to 8902 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read 000005[1] 1779068903.396011 mapwrite 0x4be313 thru 0x4cd284 (0xef72 bytes) truncating to largest ever: 0x6fabc 000006[1] 1779068903.648258 trunc from 0x4cd285 to 0x06fabb (0x45d7c9 bytes) 000007[0] 1779068903.723582 write 0x1a6b09 thru 0x1ae4ff (0x79f7 bytes) 000008[0] 1779068903.768908 read 0x18ac61 thru 0x195253 (0xa5f3 bytes) 000009[0] 1779068903.800882 read 0x00986f thru 0x00bc5e (0x23f0 bytes) truncating to largest ever: 0x9491ce 000010[1] 1779068903.855177 trunc from 0x1ae500 to 0x9491cd (0x79acce bytes) 000011[1] 1779068903.904551 trunc from 0x9491ce to 0x2e381e (0x6659af bytes) 000016[1] 1779068903.960231 mapwrite 0x7c899d thru 0x7d79e9 (0xf04d bytes) 000019[0] 1779068904.097736 mapread 0x431b9a thru 0x432c13 (0x107a bytes) 000020[1] 1779068904.140844 trunc from 0x7d79ea to 0x3ff033 (0x3d89b6 bytes) 000023[0] 1779068904.194367 mapwrite 0x8b1c1c thru 0x8b8d31 (0x7116 bytes) 000024[0] 1779068904.459187 mapread 0x305184 thru 0x313e18 (0xec95 bytes) 000027[0] 1779068904.513662 mapread 0x09ba13 thru 0x09f847 (0x3e35 bytes) 000029[1] 1779068904.538131 mapwrite 0x93d713 thru 0x9444d6 (0x6dc4 bytes) 000030[1] 1779068904.697486 mapread 0x25a283 thru 0x268e79 (0xebf7 bytes) 000035[1] 1779068904.847259 read 0x898f04 thru 0x89b0a8 (0x21a5 bytes) 000039[1] 1779068904.876618 read 0x8993b1 thru 0x8a215d (0x8dad bytes) 000040[1] 1779068904.960104 read 0x32fc95 thru 0x33071a (0xa86 bytes) 000041[1] 1779068905.016648 mapwrite 0x1777c1 thru 0x17e7d2 (0x7012 bytes) 000045[0] 1779068905.257595 mapwrite 0x79b864 thru 0x79ddcc (0x2569 bytes) 000047[1] 1779068905.484297 mapread 0x5c958f thru 0x5cf9d3 (0x6445 bytes) 000048[1] 1779068905.542114 read 0x926bac thru 0x92c84b (0x5ca0 bytes) 000049[0] 1779068905.552831 mapwrite 0x14b7f8 thru 0x1543fa (0x8c03 bytes) 000050[0] 1779068905.690967 trunc from 0x9444d7 to 0x5945fc (0x3afeda bytes) 000051[0] 1779068905.725165 mapwrite 0x72d6ec thru 0x73a74b (0xd060 bytes) 000052[1] 1779068905.808934 mapread 0x26b102 thru 0x278244 (0xd143 bytes) 000053[1] 1779068905.870694 mapread 0x62f2d7 thru 0x632311 (0x303b bytes) 000054[0] 1779068905.911518 mapwrite 0x68ddf9 thru 0x69cbcf (0xedd7 bytes) 000055[1] 1779068906.006897 mapread 0x44b535 thru 0x458c19 (0xd6e5 bytes) 000056[1] 1779068906.057793 mapread 0x1e02ef thru 0x1e7fac (0x7cbe bytes) 000057[0] 1779068906.076666 mapread 0x4c65cc thru 0x4c7016 (0xa4b bytes) 000059[1] 1779068906.086315 mapread 0x2c7abf thru 0x2d35be (0xbb00 bytes) truncating to largest ever: 0x95212d 000060[0] 1779068906.120037 trunc from 0x73a74c to 0x95212c (0x2179e1 bytes) 000061[1] 1779068906.202595 write 0x4ff574 thru 0x50f341 (0xfdce bytes) 000064[1] 1779068906.274052 read 0x3fbb21 thru 0x408791 (0xcc71 bytes) 000067[0] 1779068906.315559 trunc from 0x95212d to 0x80ebd7 (0x143555 bytes) 000068[1] 1779068906.401906 mapread 0x73ffac thru 0x74d982 (0xd9d7 bytes) 000070[1] 1779068906.480861 trunc from 0x80ebd8 to 0x5da2d9 (0x2348fe bytes) 000072[1] 1779068906.593166 read 0x599b31 thru 0x5a172e (0x7bfe bytes) 000075[1] 1779068906.663095 trunc from 0x5da2da to 0x74f863 (0x17558a bytes) 000078[0] 1779068906.753386 mapwrite 0x1d888e thru 0x1db59d (0x2d10 bytes) 000079[1] 1779068906.855036 mapread 0x68e361 thru 0x698844 (0xa4e4 bytes) 000081[1] 1779068906.907324 mapwrite 0x34103c thru 0x34e2b1 (0xd276 bytes) 000083[0] 1779068907.006225 read 0x6c508d thru 0x6cdbb3 (0x8b27 bytes) 000084[0] 1779068907.042105 trunc from 0x74f864 to 0x942a45 (0x1f31e2 bytes) 000085[1] 1779068907.087532 mapwrite 0x26b25e thru 0x27584d (0xa5f0 bytes) 000086[1] 1779068907.182946 write 0x7eaf4a thru 0x7f07ca (0x5881 bytes) 000087[1] 1779068907.223880 trunc from 0x942a46 to 0x757b34 (0x1eaf11 bytes) 000088[1] 1779068907.276503 mapread 0x36795a thru 0x37746a (0xfb11 bytes) truncating to largest ever: 0x980268 000090[1] 1779068907.304580 trunc from 0x757b35 to 0x980267 (0x228733 bytes) 000091[0] 1779068907.333314 read 0x483e8a thru 0x48e8e4 (0xaa5b bytes) 000092[0] 1779068907.357850 trunc from 0x980268 to 0x42eade (0x551789 bytes) 000093[0] 1779068907.400244 mapread 0x2f2553 thru 0x2fe7e5 (0xc293 bytes) 000094[1] 1779068907.450920 trunc from 0x42eadf to 0x182b69 (0x2abf75 bytes) 000097[0] 1779068907.542490 mapread 0x08f2f7 thru 0x09346b (0x4175 bytes) 000099[0] 1779068907.601594 trunc from 0x182b6a to 0x300eda (0x17e371 bytes) 000100[1] 1779068907.647715 mapread 0x08acc7 thru 0x08d8f9 (0x2c33 bytes) 000101[0] 1779068907.702397 mapwrite 0x8f5e82 thru 0x8f739a (0x1519 bytes) 000102[1] 1779068907.761294 write 0x35e9dc thru 0x360394 (0x19b9 bytes) 000103[1] 1779068907.808390 mapwrite 0x2b55f6 thru 0x2c3eb9 (0xe8c4 bytes) 000106[0] 1779068907.959566 trunc from 0x8f739b to 0x12e29c (0x7c90fe bytes) 000116[0] 1779068908.125180 read 0x0526e5 thru 0x06160d (0xef29 bytes) truncating to largest ever: 0x9c44cc 000121[0] 1779068908.191329 trunc from 0x12e29d to 0x9c44cb (0x89622f bytes) 000125[0] 1779068908.285082 write 0x8c40e6 thru 0x8c5c35 (0x1b50 bytes) 000127[0] 1779068908.452662 mapwrite 0x6950c7 thru 0x6a3f6a (0xeea4 bytes) 000130[0] 1779068908.546126 trunc from 0x9c44cc to 0x8e50fe (0xdf3cd bytes) 000133[1] 1779068908.574710 read 0x122a7f thru 0x128273 (0x57f5 bytes) 000136[1] 1779068908.650825 read 0x36815b thru 0x36cb7b (0x4a21 bytes) 000137[0] 1779068908.658758 trunc from 0x8e50ff to 0x1cb877 (0x719887 bytes) 000140[1] 1779068908.748843 trunc from 0x1cb878 to 0x4f98d9 (0x32e062 bytes) 000143[1] 1779068908.795326 mapread 0x4ee555 thru 0x4f98d9 (0xb385 bytes) 000145[0] 1779068908.871276 read 0x241cc1 thru 0x2453d7 (0x3717 bytes) 000148[0] 1779068908.968769 mapwrite 0x1832f8 thru 0x18997d (0x6686 bytes) 000149[0] 1779068909.112886 mapread 0x2f7025 thru 0x2fd197 (0x6173 bytes) 000150[1] 1779068909.159195 mapwrite 0x92e4ae thru 0x93c869 (0xe3bc bytes) 000151[1] 1779068909.370202 write 0x8bbae2 thru 0x8bc55b (0xa7a bytes) 000154[1] 1779068909.468472 mapwrite 0x4ae00f thru 0x4b277d (0x476f bytes) 000155[0] 1779068909.598713 mapread 0x45a9d1 thru 0x45ee0a (0x443a bytes) 000157[0] 1779068909.675505 mapwrite 0x2254f5 thru 0x2260d2 (0xbde bytes) 000158[0] 1779068909.758765 mapread 0x339e6f thru 0x34857b (0xe70d bytes) 000161[1] 1779068909.824109 read 0x327af4 thru 0x332fe3 (0xb4f0 bytes) 000162[1] 1779068909.856789 trunc from 0x93c86a to 0x7f022f (0x14c63a bytes) 000163[1] 1779068909.900910 mapwrite 0x481b75 thru 0x490965 (0xedf1 bytes) 000164[1] 1779068909.974321 mapread 0x31b14a thru 0x329e8e (0xed45 bytes) 000166[1] 1779068910.025132 mapread 0x6b0414 thru 0x6b5761 (0x534e bytes) 000170[1] 1779068910.087840 mapread 0x0ab17b thru 0x0b7ca0 (0xcb26 bytes) 000171[0] 1779068910.140125 mapread 0x17b9a9 thru 0x18a34d (0xe9a5 bytes) 000172[1] 1779068910.189446 trunc from 0x7f0230 to 0x16468a (0x68bba5 bytes) 000173[1] 1779068910.222461 mapread 0x102cb3 thru 0x10e808 (0xbb56 bytes) 000174[1] 1779068910.259773 read 0x0394b5 thru 0x0434c2 (0xa00e bytes) 000175[1] 1779068910.320024 mapwrite 0x5d6f93 thru 0x5da1cd (0x323b bytes) 000178[1] 1779068910.547959 write 0x1ec21f thru 0x1f09b2 (0x4794 bytes) 000180[0] 1779068910.549157 mapwrite 0x943728 thru 0x94a249 (0x6b22 bytes) 000182[0] 1779068910.726605 mapread 0x7e01f3 thru 0x7f0111 (0xff1f bytes) 000185[1] 1779068910.785169 write 0x831229 thru 0x835b06 (0x48de bytes) 000188[0] 1779068910.856480 write 0x7c4d9b thru 0x7d19b8 (0xcc1e bytes) 000190[1] 1779068910.899086 mapwrite 0x1cd100 thru 0x1d1012 (0x3f13 bytes) 000191[0] 1779068910.981311 read 0x675444 thru 0x682623 (0xd1e0 bytes) 000192[0] 1779068911.004932 mapwrite 0x657624 thru 0x661f77 (0xa954 bytes) 000194[0] 1779068911.098455 trunc from 0x94a24a to 0x1d3dda (0x77646f bytes) 000195[0] 1779068911.184848 write 0x6657e0 thru 0x66a73f (0x4f60 bytes) 000196[1] 1779068911.221020 trunc from 0x66a740 to 0x1afcc9 (0x4baa76 bytes) 000197[1] 1779068911.297666 mapwrite 0x1a3178 thru 0x1ad67a (0xa503 bytes) truncating to largest ever: 0x9df446 000198[1] 1779068911.479723 trunc from 0x1afcca to 0x9df445 (0x82f77c bytes) 000200[0] 1779068911.508866 mapread 0x872e8d thru 0x876c83 (0x3df7 bytes) 000202[0] 1779068911.645646 write 0x34629f thru 0x35459d (0xe2ff bytes) 000207[1] 1779068911.769696 trunc from 0x9df446 to 0x408243 (0x5d7202 bytes) 000211[0] 1779068911.881824 write 0x1895b7 thru 0x191818 (0x8262 bytes) 000212[0] 1779068911.918009 trunc from 0x408244 to 0x7177c6 (0x30f583 bytes) 000213[1] 1779068911.998964 write 0x07715c thru 0x077dde (0xc83 bytes) 000215[0] 1779068912.087284 mapwrite 0x9fa83f thru 0x9fffff (0x57c1 bytes) 000216[0] 1779068912.311984 write 0x569fbe thru 0x579642 (0xf685 bytes) 000217[0] 1779068912.329502 write 0x998743 thru 0x99a26f (0x1b2d bytes) 000221[1] 1779068912.415918 write 0x552187 thru 0x55d5fa (0xb474 bytes) 000222[1] 1779068912.461030 mapread 0x8dfd9e thru 0x8e2105 (0x2368 bytes) 000224[1] 1779068912.575275 mapwrite 0x9c50e8 thru 0x9cb1b4 (0x60cd bytes) 000225[0] 1779068912.711452 trunc from 0xa00000 to 0x0bc166 (0x943e99 bytes) 000226[1] 1779068912.789804 trunc from 0x0bc167 to 0x137df2 (0x7bc8c bytes) 000227[1] 1779068912.837422 mapread 0x069371 thru 0x06cc5e (0x38ee bytes) 000229[0] 1779068912.911746 write 0x07c5a8 thru 0x08af90 (0xe9e9 bytes) 000230[1] 1779068912.961466 trunc from 0x137df3 to 0x129482 (0xe970 bytes) 000233[1] 1779068913.213387 read 0x08d41c thru 0x08e266 (0xe4b bytes) 000234[1] 1779068913.302065 mapwrite 0x6a3a9b thru 0x6ad890 (0x9df6 bytes) 000235[1] 1779068913.533225 read 0x255a17 thru 0x261205 (0xb7ef bytes) 000238[0] 1779068913.587665 read 0x38ee28 thru 0x38f8b5 (0xa8e bytes) 000239[0] 1779068913.640215 trunc from 0x6ad891 to 0x5400b8 (0x16d7d8 bytes) 000240[0] 1779068913.687665 trunc from 0x5400b9 to 0x1a49d4 (0x39b6e4 bytes) 000241[1] 1779068913.710494 write 0x10ff84 thru 0x11e964 (0xe9e1 bytes) 000242[0] 1779068913.754324 trunc from 0x1a49d5 to 0x857315 (0x6b2941 bytes) 000247[1] 1779068913.916958 trunc from 0x857316 to 0x1ce050 (0x6892c5 bytes) 000249[1] 1779068913.973782 write 0x33adb0 thru 0x33f78d (0x49de bytes) 000250[1] 1779068914.019138 trunc from 0x33f78e to 0x283c88 (0xbbb05 bytes) 000251[1] 1779068914.074051 mapwrite 0x547784 thru 0x551cc6 (0xa543 bytes) 000252[0] 1779068914.183055 mapread 0x075041 thru 0x07efb9 (0x9f79 bytes) 000253[1] 1779068914.228112 read 0x071071 thru 0x074423 (0x33b3 bytes) 000254[0] 1779068914.250015 write 0x5cd4b9 thru 0x5da136 (0xcc7e bytes) 000255[0] 1779068914.265891 trunc from 0x5da137 to 0x6d6287 (0xfc151 bytes) 000256[0] 1779068914.322227 read 0x47c885 thru 0x47d82b (0xfa7 bytes) 000257[1] 1779068914.331275 write 0x461894 thru 0x46fa25 (0xe192 bytes) 000259[1] 1779068914.384687 read 0x47fd7b thru 0x48c8f1 (0xcb77 bytes) 000263[0] 1779068914.448978 mapwrite 0x982bd8 thru 0x98d7f8 (0xac21 bytes) 000268[1] 1779068914.638234 write 0x8e9f6a thru 0x8f33e2 (0x9479 bytes) 000273[0] 1779068914.725922 write 0x186ae2 thru 0x18d56f (0x6a8e bytes) 000275[1] 1779068914.757819 mapread 0x84dccb thru 0x853219 (0x554f bytes) 000276[0] 1779068914.796225 mapread 0x6f7508 thru 0x7033a3 (0xbe9c bytes) 000277[0] 1779068914.842206 read 0x716dd1 thru 0x7252c4 (0xe4f4 bytes) 000279[0] 1779068914.874945 mapwrite 0x3a7f29 thru 0x3b001d (0x80f5 bytes) 000281[0] 1779068914.969683 mapread 0x1b2ebd thru 0x1c2be3 (0xfd27 bytes) 000283[0] 1779068914.999114 mapread 0x35f924 thru 0x36bc09 (0xc2e6 bytes) 000284[1] 1779068915.029631 write 0x59eb40 thru 0x5addf3 (0xf2b4 bytes) 000289[0] 1779068915.126166 trunc from 0x98d7f9 to 0x3c2380 (0x5cb478 bytes) 000290[1] 1779068915.169114 trunc from 0x3c2381 to 0x4748e0 (0xb2560 bytes) 000291[0] 1779068915.244541 read 0x0f6f7b thru 0x0fb0dd (0x4163 bytes) 000293[1] 1779068915.291637 mapread 0x34a7d7 thru 0x3581e2 (0xda0c bytes) 000297[1] 1779068915.327674 trunc from 0x4748e1 to 0x1421c1 (0x33271f bytes) 000298[1] 1779068915.355181 read 0x019b6b thru 0x021e40 (0x82d6 bytes) 000301[1] 1779068915.437593 mapwrite 0x1869bf thru 0x19585f (0xeea1 bytes) 000302[0] 1779068915.527298 mapwrite 0x560335 thru 0x567ef0 (0x7bbc bytes) 000305[1] 1779068915.681238 trunc from 0x567ef1 to 0x741a47 (0x1d9b57 bytes) 000306[1] 1779068915.720017 mapwrite 0x5ce7ff thru 0x5cfa14 (0x1216 bytes) 000307[0] 1779068915.887926 mapwrite 0x0ea2c9 thru 0x0f6ac7 (0xc7ff bytes) 000308[0] 1779068915.990496 mapwrite 0x03c656 thru 0x04a1c3 (0xdb6e bytes) 000309[0] 1779068916.065197 write 0x4ca9da thru 0x4d3eb5 (0x94dc bytes) 000311[1] 1779068916.122768 trunc from 0x741a48 to 0x237bc1 (0x509e86 bytes) 000314[1] 1779068916.210184 read 0x1b21dd thru 0x1b9941 (0x7765 bytes) 000315[1] 1779068916.306602 write 0x1b616e thru 0x1ba81f (0x46b2 bytes) 000316[0] 1779068916.307793 mapread 0x08d7e8 thru 0x094720 (0x6f39 bytes) 000317[0] 1779068916.374981 trunc from 0x237bc2 to 0x080315 (0x1b78ac bytes) 000320[0] 1779068916.478461 write 0x607e41 thru 0x61547c (0xd63c bytes) 000324[1] 1779068916.527937 mapread 0x066f07 thru 0x07055d (0x9657 bytes) 000326[1] 1779068916.625042 trunc from 0x61547d to 0x550811 (0xc4c6b bytes) 000330[1] 1779068916.742661 mapwrite 0x58dd25 thru 0x596469 (0x8745 bytes) 000331[0] 1779068916.907211 trunc from 0x59646a to 0x6bf6d9 (0x129270 bytes) 000332[0] 1779068916.976721 mapread 0x36e20a thru 0x37c346 (0xe13d bytes) 000333[1] 1779068917.050449 write 0x6cd857 thru 0x6cdb32 (0x2dc bytes) 000334[0] 1779068917.086176 read 0x17ef25 thru 0x188b6f (0x9c4b bytes) 000339[1] 1779068917.182940 mapwrite 0x422a83 thru 0x429357 (0x68d5 bytes) 000342[1] 1779068917.494469 trunc from 0x6cdb33 to 0x2e419d (0x3e9995 bytes) 000343[0] 1779068917.585882 mapwrite 0x90d0f4 thru 0x90ec06 (0x1b13 bytes) 000347[1] 1779068917.833071 write 0x9fb12e thru 0x9fcba6 (0x1a79 bytes) 000349[1] 1779068917.880296 write 0x9855f9 thru 0x987573 (0x1f7b bytes) 000351[0] 1779068917.940957 read 0x7d8950 thru 0x7deeff (0x65b0 bytes) 000352[0] 1779068917.961797 mapwrite 0x1090dc thru 0x10f8b6 (0x67db bytes) 000353[0] 1779068918.074270 read 0x5b1683 thru 0x5be9a4 (0xd322 bytes) 000355[0] 1779068918.132138 trunc from 0x9fcba7 to 0x95e224 (0x9e982 bytes) 000356[0] 1779068918.156628 mapwrite 0x8afda8 thru 0x8b01fd (0x456 bytes) 000357[0] 1779068918.195935 write 0x260fa5 thru 0x2697e7 (0x8843 bytes) 000360[0] 1779068918.265755 write 0x8d7edb thru 0x8e4a22 (0xcb48 bytes) 000361[0] 1779068918.282291 write 0x4eed86 thru 0x4fe2d2 (0xf54d bytes) 000363[1] 1779068918.328220 write 0x9a09ca thru 0x9a0b5e (0x195 bytes) 000364[0] 1779068918.370156 write 0x647cfa thru 0x64968f (0x1996 bytes) 000365[0] 1779068918.402527 read 0x24c287 thru 0x258aff (0xc879 bytes) 000367[1] 1779068918.442064 write 0x661a56 thru 0x66ab2d (0x90d8 bytes) 000372[1] 1779068918.558229 trunc from 0x9a0b5f to 0x442621 (0x55e53d bytes) 000377[1] 1779068918.749442 mapwrite 0x6ea450 thru 0x6f1e02 (0x79b3 bytes) 000379[0] 1779068918.926066 trunc from 0x6f1e03 to 0x5eb86b (0x106597 bytes) 000380[1] 1779068918.972241 write 0x4f7a84 thru 0x5056bd (0xdc3a bytes) 000383[1] 1779068919.094933 write 0x31e6a6 thru 0x328e91 (0xa7ec bytes) 000384[0] 1779068919.147897 mapread 0x33b1e3 thru 0x347e42 (0xcc60 bytes) 000385[1] 1779068919.235913 mapread 0x1093bc thru 0x117955 (0xe59a bytes) 000389[1] 1779068919.408707 mapread 0x5538d4 thru 0x5614aa (0xdbd7 bytes) 000390[1] 1779068919.595324 write 0x30f656 thru 0x31f10b (0xfab6 bytes) 000392[1] 1779068919.716200 mapwrite 0x1396d8 thru 0x144a4e (0xb377 bytes) 000393[1] 1779068919.886657 read 0x489bba thru 0x498ee5 (0xf32c bytes) 000394[1] 1779068919.915324 read 0x4c1cef thru 0x4c3277 (0x1589 bytes) 000397[0] 1779068919.973104 trunc from 0x5eb86c to 0x7e14b1 (0x1f5c46 bytes) 000398[0] 1779068920.040027 write 0x04653b thru 0x05561d (0xf0e3 bytes) 000399[0] 1779068920.066719 read 0x4d00d8 thru 0x4d25cc (0x24f5 bytes) 000400[0] 1779068920.090068 read 0x317a1d thru 0x3256bd (0xdca1 bytes) 000401[1] 1779068920.105706 mapwrite 0x71b332 thru 0x71e37c (0x304b bytes) 000402[1] 1779068920.220527 mapwrite 0x2a354f thru 0x2a5989 (0x243b bytes) 000405[0] 1779068920.351759 mapread 0x1b4581 thru 0x1b8d45 (0x47c5 bytes) 000406[1] 1779068920.399697 mapread 0x502be2 thru 0x50c900 (0x9d1f bytes) 000410[1] 1779068920.470713 write 0x259120 thru 0x2652cf (0xc1b0 bytes) 000412[0] 1779068920.525237 read 0x56086b thru 0x56be7e (0xb614 bytes) 000414[0] 1779068920.572323 mapread 0x2e4e5d thru 0x2efe0d (0xafb1 bytes) 000416[0] 1779068920.664405 trunc from 0x7e14b2 to 0x8a5fd8 (0xc4b27 bytes) 000417[0] 1779068920.720750 trunc from 0x8a5fd9 to 0x13f705 (0x7668d3 bytes) 000420[1] 1779068920.739472 mapread 0x10ad65 thru 0x1182a4 (0xd540 bytes) 000421[1] 1779068920.796900 mapwrite 0x10808b thru 0x10d2c6 (0x523c bytes) 000424[0] 1779068921.013654 mapread 0x01a685 thru 0x020a6b (0x63e7 bytes) 000426[0] 1779068921.104954 read 0x0a2ac8 thru 0x0a3fbd (0x14f6 bytes) 000427[0] 1779068921.158511 read 0x0f2c5e thru 0x0f8c43 (0x5fe6 bytes) 000428[1] 1779068921.159764 trunc from 0x13f706 to 0x064fbf (0xda746 bytes) 000430[1] 1779068921.225568 write 0x0a5851 thru 0x0b0556 (0xad06 bytes) 000432[0] 1779068921.229556 read 0x0a4404 thru 0x0a9b2e (0x572b bytes) 000433[0] 1779068921.283298 read 0x0631f9 thru 0x0678ec (0x46f4 bytes) 000435[1] 1779068921.310743 read 0x0889b5 thru 0x08c4e8 (0x3b34 bytes) 000437[0] 1779068921.346875 mapwrite 0x598f0d thru 0x5a41e3 (0xb2d7 bytes) 000438[1] 1779068921.549762 write 0x07c748 thru 0x0841e1 (0x7a9a bytes) 000440[1] 1779068921.651780 mapwrite 0x11ed5a thru 0x1295fd (0xa8a4 bytes) 000442[0] 1779068921.811131 write 0x555553 thru 0x561009 (0xbab7 bytes) 000443[0] 1779068921.848716 read 0x037a1e thru 0x039e80 (0x2463 bytes) 000444[0] 1779068921.895485 read 0x499f52 thru 0x4a1c46 (0x7cf5 bytes) 000445[0] 1779068921.910255 write 0x6f23e0 thru 0x6fd42c (0xb04d bytes) 000447[1] 1779068921.944584 mapwrite 0x08096f thru 0x08b4df (0xab71 bytes) 000449[1] 1779068922.112566 trunc from 0x6fd42d to 0x8f882b (0x1fb3ff bytes) 000451[1] 1779068922.175308 mapwrite 0x448f15 thru 0x452aa3 (0x9b8f bytes) 000452[0] 1779068922.345197 read 0x253be4 thru 0x262d8a (0xf1a7 bytes) 000454[0] 1779068922.392722 mapread 0x1f8a0e thru 0x205033 (0xc626 bytes) 000455[1] 1779068922.423666 read 0x66cce1 thru 0x675464 (0x8784 bytes) 000456[1] 1779068922.434063 mapread 0x41e8a6 thru 0x424bf6 (0x6351 bytes) 000457[1] 1779068922.448303 mapwrite 0x3436d0 thru 0x3507c4 (0xd0f5 bytes) 000462[0] 1779068922.598646 trunc from 0x8f882c to 0x2a9f67 (0x64e8c4 bytes) 000468[1] 1779068922.738409 mapwrite 0x8fb3ae thru 0x904560 (0x91b3 bytes) 000470[1] 1779068922.919223 trunc from 0x904561 to 0x6d3970 (0x230bf0 bytes) 000472[0] 1779068922.970946 mapwrite 0x944f42 thru 0x94da1f (0x8ade bytes) 000476[0] 1779068923.117304 read 0x91adb8 thru 0x92529b (0xa4e4 bytes) 000478[1] 1779068923.128845 trunc from 0x94da20 to 0x09cfc8 (0x8b0a57 bytes) 000480[0] 1779068923.175919 trunc from 0x09cfc9 to 0x4fba73 (0x45eaab bytes) 000484[0] 1779068923.303117 mapwrite 0x5e5631 thru 0x5ec287 (0x6c57 bytes) 000487[1] 1779068923.455437 mapwrite 0x443c65 thru 0x44973e (0x5ada bytes) 000490[1] 1779068923.581241 read 0x15ba11 thru 0x1614f3 (0x5ae3 bytes) 000491[0] 1779068923.602238 mapread 0x0f0e81 thru 0x0fb549 (0xa6c9 bytes) 000492[0] 1779068923.634006 mapread 0x477103 thru 0x478188 (0x1086 bytes) 000493[1] 1779068923.669199 read 0x571e27 thru 0x57d451 (0xb62b bytes) 000494[1] 1779068923.678736 mapwrite 0x703df0 thru 0x713615 (0xf826 bytes) 000495[0] 1779068923.843142 mapwrite 0x96320c thru 0x972bba (0xf9af bytes) 000496[1] 1779068923.985062 mapwrite 0x3619f7 thru 0x369523 (0x7b2d bytes) 000498[0] 1779068924.087869 write 0x789ae3 thru 0x78b240 (0x175e bytes) 000499[1] 1779068924.111976 write 0x5991d9 thru 0x5a4c1f (0xba47 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 8924 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000007[1] 1779068925.030296 write_OD 0x3ea000 thru 0x3f1fff (0x8000 bytes) 000008[0] 1779068925.097128 write_OD 0x36c000 thru 0x36ffff (0x4000 bytes) 000009[1] 1779068925.147821 mapwrite 0x684000 thru 0x684f83 (0xf84 bytes) 000011[1] 1779068925.215685 mapread 0x270000 thru 0x2771ad (0x71ae bytes) 000013[0] 1779068925.253631 mapwrite 0x30b000 thru 0x316641 (0xb642 bytes) 000017[1] 1779068925.394872 write_OD 0x682000 thru 0x684fff (0x3000 bytes) 000018[0] 1779068925.443698 write_OD 0x40d000 thru 0x418fff (0xc000 bytes) 000019[0] 1779068925.532900 write_OD 0x4c9000 thru 0x4d2fff (0xa000 bytes) 000021[1] 1779068925.598593 read_OD 0x4b4000 thru 0x4b6fff (0x3000 bytes) 000022[0] 1779068925.607694 write_OD 0x1b0000 thru 0x1b0fff (0x1000 bytes) truncating to largest ever: 0x154f30 000024[0] 1779068925.681212 trunc from 0x685000 to 0x154f2f (0x5300d0 bytes) 000025[0] 1779068925.738806 mapread 0x13e000 thru 0x14a1c4 (0xc1c5 bytes) truncating to largest ever: 0x8f6950 000026[0] 1779068925.785115 trunc from 0x154f30 to 0x8f694f (0x7a1a20 bytes) 000032[1] 1779068925.824700 trunc from 0x8f6950 to 0x52ae8d (0x3cbac2 bytes) 000033[0] 1779068925.856221 mapwrite 0x58a000 thru 0x5918d2 (0x78d3 bytes) 000035[1] 1779068925.973114 write_OD 0x019000 thru 0x026fff (0xe000 bytes) 000036[0] 1779068926.025952 write_OD 0x77c000 thru 0x78afff (0xf000 bytes) 000037[0] 1779068926.076717 mapread 0x00b000 thru 0x01472c (0x972d bytes) 000038[0] 1779068926.091136 trunc from 0x78b000 to 0x83fd06 (0xb4d07 bytes) 000049[0] 1779068926.249122 trunc from 0x83fd07 to 0x2553ab (0x5ea95b bytes) 000052[0] 1779068926.350803 write_OD 0x8d5000 thru 0x8e1fff (0xd000 bytes) 000053[1] 1779068926.485736 mapread 0x4e1000 thru 0x4e1397 (0x398 bytes) 000061[1] 1779068926.495783 write_OD 0x7dc000 thru 0x7e1fff (0x6000 bytes) truncating to largest ever: 0x9af301 000065[1] 1779068926.554748 trunc from 0x8e2000 to 0x9af300 (0xcd301 bytes) 000067[1] 1779068926.590800 mapwrite 0x5d8000 thru 0x5e5b7c (0xdb7d bytes) 000071[0] 1779068926.701072 mapwrite 0x9b5000 thru 0x9bbbec (0x6bed bytes) 000072[1] 1779068926.809318 mapwrite 0x011000 thru 0x0195b2 (0x85b3 bytes) 000073[0] 1779068926.888286 trunc from 0x9bbbed to 0x32771f (0x6944cd bytes) 000074[0] 1779068926.917270 read_OD 0x1b6000 thru 0x1c3fff (0xe000 bytes) 000077[0] 1779068926.944412 mapwrite 0x3c0000 thru 0x3ccd3c (0xcd3d bytes) 000079[0] 1779068927.170837 mapwrite 0x8d4000 thru 0x8d9360 (0x5361 bytes) 000082[1] 1779068927.353930 trunc from 0x8d9361 to 0x45834c (0x481014 bytes) 000083[0] 1779068927.388989 mapwrite 0x30a000 thru 0x3127ca (0x87cb bytes) 000084[1] 1779068927.485580 read_OD 0x1d8000 thru 0x1d8fff (0x1000 bytes) 000086[0] 1779068927.499437 mapwrite 0x981000 thru 0x990f6f (0xff70 bytes) 000090[1] 1779068927.702649 write_OD 0x73c000 thru 0x748fff (0xd000 bytes) 000092[0] 1779068927.790665 write_OD 0x09e000 thru 0x0a0fff (0x3000 bytes) 000094[1] 1779068927.842485 read_OD 0x26c000 thru 0x26cfff (0x1000 bytes) 000095[0] 1779068927.850545 read_OD 0x434000 thru 0x43ffff (0xc000 bytes) 000097[0] 1779068927.865711 mapwrite 0x925000 thru 0x932cfa (0xdcfb bytes) 000098[1] 1779068927.980319 trunc from 0x990f70 to 0x2e1690 (0x6af8df bytes) 000101[0] 1779068928.063222 write_OD 0x2f7000 thru 0x2f8fff (0x2000 bytes) 000103[1] 1779068928.116071 trunc from 0x2f9000 to 0x85dc06 (0x564c07 bytes) 000104[1] 1779068928.149695 mapread 0x855000 thru 0x85c94e (0x794f bytes) 000109[0] 1779068928.244025 mapread 0x534000 thru 0x53a275 (0x6276 bytes) 000111[1] 1779068928.289176 read_OD 0x100000 thru 0x10afff (0xb000 bytes) 000112[1] 1779068928.318055 read_OD 0x31d000 thru 0x322fff (0x6000 bytes) 000113[0] 1779068928.324972 mapwrite 0x57c000 thru 0x58696f (0xa970 bytes) 000114[1] 1779068928.442207 mapwrite 0x735000 thru 0x73f51d (0xa51e bytes) 000117[0] 1779068928.549043 write_OD 0x5fc000 thru 0x601fff (0x6000 bytes) 000122[0] 1779068928.599918 mapwrite 0x247000 thru 0x255404 (0xe405 bytes) 000126[1] 1779068928.850043 read_OD 0x562000 thru 0x563fff (0x2000 bytes) 000127[0] 1779068928.872921 mapwrite 0x280000 thru 0x289e9f (0x9ea0 bytes) 000128[0] 1779068928.964668 trunc from 0x85dc07 to 0x1446c7 (0x71953f bytes) 000130[1] 1779068929.020668 write_OD 0x5b6000 thru 0x5b9fff (0x4000 bytes) 000132[1] 1779068929.079249 mapread 0x0fa000 thru 0x101ea8 (0x7ea9 bytes) 000140[1] 1779068929.094224 write_OD 0x6ed000 thru 0x6f1fff (0x5000 bytes) 000142[1] 1779068929.142291 trunc from 0x6f2000 to 0x093cd5 (0x65e32a bytes) 000144[0] 1779068929.177561 mapwrite 0x4a4000 thru 0x4ab476 (0x7477 bytes) 000146[1] 1779068929.277240 mapread 0x173000 thru 0x1779eb (0x49ec bytes) 000147[1] 1779068929.312951 read_OD 0x15e000 thru 0x160fff (0x3000 bytes) 000148[1] 1779068929.333289 read_OD 0x237000 thru 0x244fff (0xe000 bytes) 000153[1] 1779068929.346096 mapread 0x19e000 thru 0x1a8243 (0xa244 bytes) 000154[0] 1779068929.370379 read_OD 0x210000 thru 0x21bfff (0xc000 bytes) 000155[0] 1779068929.378308 mapwrite 0x17e000 thru 0x186606 (0x8607 bytes) 000156[1] 1779068929.454356 mapwrite 0x266000 thru 0x26cb6f (0x6b70 bytes) 000157[0] 1779068929.581134 mapread 0x0b4000 thru 0x0b544b (0x144c bytes) 000160[0] 1779068929.617653 mapwrite 0x554000 thru 0x56028e (0xc28f bytes) 000161[0] 1779068929.813762 mapread 0x4cb000 thru 0x4d5c98 (0xac99 bytes) 000163[0] 1779068929.871618 write_OD 0x76a000 thru 0x76cfff (0x3000 bytes) 000164[0] 1779068929.969781 mapread 0x37c000 thru 0x38b3f1 (0xf3f2 bytes) 000165[0] 1779068930.020082 mapwrite 0x197000 thru 0x1a615f (0xf160 bytes) 000168[1] 1779068930.115725 mapwrite 0x604000 thru 0x608953 (0x4954 bytes) 000172[0] 1779068930.282232 write_OD 0x501000 thru 0x508fff (0x8000 bytes) 000173[0] 1779068930.363129 mapread 0x762000 thru 0x76c620 (0xa621 bytes) 000174[0] 1779068930.444222 write_OD 0x59b000 thru 0x59bfff (0x1000 bytes) 000176[1] 1779068930.553138 mapwrite 0x7ad000 thru 0x7b02e2 (0x32e3 bytes) 000178[1] 1779068930.794757 mapwrite 0x904000 thru 0x9112f2 (0xd2f3 bytes) 000179[0] 1779068930.966115 trunc from 0x9112f3 to 0x108385 (0x808f6d bytes) 000180[0] 1779068930.998030 trunc from 0x108386 to 0x69b3fa (0x593075 bytes) 000185[1] 1779068931.065810 mapread 0x296000 thru 0x29705d (0x105e bytes) 000186[1] 1779068931.095117 read_OD 0x0db000 thru 0x0e0fff (0x6000 bytes) 000189[1] 1779068931.100522 mapread 0x37b000 thru 0x37fdb8 (0x4db9 bytes) 000192[1] 1779068931.121418 mapwrite 0x559000 thru 0x5627f3 (0x97f4 bytes) 000195[1] 1779068931.222398 trunc from 0x69b3fb to 0x28eb42 (0x40c8b8 bytes) 000196[0] 1779068931.245146 write_OD 0x5b7000 thru 0x5b8fff (0x2000 bytes) 000198[1] 1779068931.305301 trunc from 0x5b9000 to 0x71f7bc (0x1667bd bytes) 000202[1] 1779068931.483277 mapwrite 0x7e4000 thru 0x7e783d (0x383e bytes) 000204[1] 1779068931.697825 write_OD 0x596000 thru 0x59efff (0x9000 bytes) 000205[0] 1779068931.789021 write_OD 0x5a2000 thru 0x5affff (0xe000 bytes) 000207[1] 1779068931.917499 write_OD 0x7d4000 thru 0x7e2fff (0xf000 bytes) 000211[1] 1779068932.017862 write_OD 0x21f000 thru 0x22cfff (0xe000 bytes) 000213[0] 1779068932.088137 mapread 0x644000 thru 0x64c0fe (0x80ff bytes) 000214[0] 1779068932.146103 write_OD 0x90a000 thru 0x915fff (0xc000 bytes) 000219[0] 1779068932.288929 read_OD 0x8fd000 thru 0x905fff (0x9000 bytes) 000223[1] 1779068932.302673 read_OD 0x158000 thru 0x159fff (0x2000 bytes) 000225[0] 1779068932.310943 trunc from 0x916000 to 0x636b86 (0x2df479 bytes) 000226[0] 1779068932.365983 mapread 0x209000 thru 0x212235 (0x9236 bytes) 000232[1] 1779068932.428258 mapread 0x4a8000 thru 0x4b15bb (0x95bc bytes) 000235[0] 1779068932.480325 trunc from 0x636b87 to 0x66ca0e (0x35e88 bytes) 000241[0] 1779068932.564285 mapwrite 0x9a3000 thru 0x9a5565 (0x2566 bytes) 000242[0] 1779068932.652258 trunc from 0x9a5566 to 0x7f7455 (0x1ae110 bytes) 000243[1] 1779068932.673047 write_OD 0x360000 thru 0x36efff (0xf000 bytes) 000244[0] 1779068932.723311 trunc from 0x7f7456 to 0x82dcfa (0x368a5 bytes) 000245[0] 1779068932.777458 read_OD 0x2ca000 thru 0x2d6fff (0xd000 bytes) 000246[1] 1779068932.795253 write_OD 0x045000 thru 0x045fff (0x1000 bytes) 000247[0] 1779068932.839201 write_OD 0x857000 thru 0x85bfff (0x5000 bytes) 000248[0] 1779068932.883127 mapwrite 0x6ee000 thru 0x6f93f2 (0xb3f3 bytes) 000250[1] 1779068932.990817 trunc from 0x85c000 to 0x0f4990 (0x76766f bytes) 000255[0] 1779068933.054721 read_OD 0x01b000 thru 0x029fff (0xf000 bytes) 000262[0] 1779068933.082829 mapread 0x09c000 thru 0x0a6a01 (0xaa02 bytes) 000271[1] 1779068933.109655 write_OD 0x8c1000 thru 0x8cdfff (0xd000 bytes) 000272[1] 1779068933.170400 write_OD 0x189000 thru 0x18ffff (0x7000 bytes) 000275[1] 1779068933.219284 write_OD 0x1ee000 thru 0x1fcfff (0xf000 bytes) 000276[1] 1779068933.279801 mapread 0x13b000 thru 0x14884f (0xd850 bytes) 000278[0] 1779068933.309598 mapread 0x151000 thru 0x153fc7 (0x2fc8 bytes) 000279[0] 1779068933.374776 mapread 0x36e000 thru 0x37c2a8 (0xe2a9 bytes) 000280[1] 1779068933.406970 mapread 0x02f000 thru 0x02ffd0 (0xfd1 bytes) 000281[1] 1779068933.420653 trunc from 0x8ce000 to 0x71ae69 (0x1b3196 bytes) 000282[0] 1779068933.465124 mapread 0x1dc000 thru 0x1dd09a (0x109b bytes) 000286[0] 1779068933.503643 write_OD 0x70b000 thru 0x70cfff (0x2000 bytes) 000289[1] 1779068933.570888 write_OD 0x53d000 thru 0x53efff (0x2000 bytes) 000293[0] 1779068933.642300 mapwrite 0x981000 thru 0x9904d8 (0xf4d9 bytes) 000294[0] 1779068933.834914 mapread 0x190000 thru 0x19cc91 (0xcc92 bytes) 000295[0] 1779068933.915159 trunc from 0x9904d9 to 0x057007 (0x9394d1 bytes) 000296[1] 1779068933.963786 write_OD 0x824000 thru 0x828fff (0x5000 bytes) 000298[0] 1779068934.021047 read_OD 0x254000 thru 0x260fff (0xd000 bytes) 000299[1] 1779068934.030505 mapread 0x1fc000 thru 0x205442 (0x9443 bytes) 000301[1] 1779068934.049127 mapread 0x748000 thru 0x749c42 (0x1c43 bytes) 000302[0] 1779068934.055441 write_OD 0x3d8000 thru 0x3ddfff (0x6000 bytes) 000303[0] 1779068934.093035 write_OD 0x7d6000 thru 0x7ddfff (0x8000 bytes) 000304[1] 1779068934.136066 mapwrite 0x406000 thru 0x41330c (0xd30d bytes) 000307[1] 1779068934.258779 write_OD 0x812000 thru 0x81bfff (0xa000 bytes) 000308[1] 1779068934.311027 mapwrite 0x174000 thru 0x1805ae (0xc5af bytes) 000309[0] 1779068934.390569 write_OD 0x13e000 thru 0x13ffff (0x2000 bytes) 000310[0] 1779068934.424109 trunc from 0x829000 to 0x99c423 (0x173424 bytes) 000311[0] 1779068934.464504 mapwrite 0x902000 thru 0x90eeb7 (0xceb8 bytes) 000312[1] 1779068934.595022 mapwrite 0x69a000 thru 0x69cf65 (0x2f66 bytes) 000314[0] 1779068934.674323 mapread 0x680000 thru 0x68fef3 (0xfef4 bytes) 000316[0] 1779068934.732580 mapwrite 0x4d9000 thru 0x4e607b (0xd07c bytes) 000320[0] 1779068934.885216 trunc from 0x99c424 to 0x8b247a (0xe9fa9 bytes) 000322[1] 1779068934.942442 trunc from 0x8b247b to 0x8dd1df (0x2ad65 bytes) 000324[0] 1779068934.985994 mapread 0x6c2000 thru 0x6c65cd (0x45ce bytes) 000325[1] 1779068935.035581 trunc from 0x8dd1e0 to 0x5ae475 (0x32ed6a bytes) 000326[0] 1779068935.075273 trunc from 0x5ae476 to 0x7cb4e2 (0x21d06d bytes) 000329[0] 1779068935.123616 mapread 0x555000 thru 0x558412 (0x3413 bytes) 000332[0] 1779068935.153320 mapwrite 0x1be000 thru 0x1bf9ba (0x19bb bytes) 000336[0] 1779068935.264141 read_OD 0x2e6000 thru 0x2e8fff (0x3000 bytes) 000337[0] 1779068935.279764 mapwrite 0x7e4000 thru 0x7e9727 (0x5728 bytes) 000338[1] 1779068935.463047 mapread 0x356000 thru 0x356e45 (0xe46 bytes) 000340[0] 1779068935.525537 trunc from 0x7e9728 to 0x0840f5 (0x765632 bytes) 000341[0] 1779068935.582851 trunc from 0x0840f6 to 0x00120a (0x82eeb bytes) 000344[1] 1779068935.636110 mapwrite 0x257000 thru 0x25bbb2 (0x4bb3 bytes) 000349[0] 1779068935.912085 write_OD 0x1bd000 thru 0x1c6fff (0xa000 bytes) 000350[0] 1779068935.990102 mapread 0x232000 thru 0x239b2b (0x7b2c bytes) 000351[0] 1779068936.021890 mapread 0x14f000 thru 0x157c82 (0x8c83 bytes) 000354[1] 1779068936.035550 write_OD 0x1c9000 thru 0x1c9fff (0x1000 bytes) 000355[0] 1779068936.125698 mapread 0x04a000 thru 0x058578 (0xe579 bytes) 000356[0] 1779068936.197133 mapread 0x24f000 thru 0x25bb20 (0xcb21 bytes) 000358[1] 1779068936.228655 mapread 0x115000 thru 0x1247c7 (0xf7c8 bytes) 000361[1] 1779068936.292930 mapwrite 0x175000 thru 0x17bc55 (0x6c56 bytes) 000365[0] 1779068936.467191 read_OD 0x15d000 thru 0x168fff (0xc000 bytes) 000366[0] 1779068936.512685 trunc from 0x25bbb3 to 0x7b36db (0x557b29 bytes) 000368[0] 1779068936.614705 read_OD 0x565000 thru 0x56afff (0x6000 bytes) 000372[1] 1779068936.659313 read_OD 0x230000 thru 0x234fff (0x5000 bytes) 000377[1] 1779068936.670537 read_OD 0x630000 thru 0x632fff (0x3000 bytes) 000380[0] 1779068936.679027 mapwrite 0x8f3000 thru 0x8ff8af (0xc8b0 bytes) 000381[1] 1779068936.937709 mapread 0x4a8000 thru 0x4abbd9 (0x3bda bytes) 000382[1] 1779068937.044959 mapwrite 0x4cd000 thru 0x4d2e45 (0x5e46 bytes) 000383[1] 1779068937.164978 write_OD 0x28d000 thru 0x292fff (0x6000 bytes) 000386[0] 1779068937.229390 read_OD 0x205000 thru 0x209fff (0x5000 bytes) 000387[0] 1779068937.248582 read_OD 0x380000 thru 0x38dfff (0xe000 bytes) 000388[0] 1779068937.257744 trunc from 0x8ff8b0 to 0x243f78 (0x6bb937 bytes) 000389[0] 1779068937.304198 trunc from 0x243f79 to 0x1ead4e (0x5922a bytes) 000390[0] 1779068937.343299 mapread 0x017000 thru 0x02180b (0xa80c bytes) 000391[1] 1779068937.399507 mapread 0x1c0000 thru 0x1cca96 (0xca97 bytes) 000392[0] 1779068937.428854 mapwrite 0x114000 thru 0x117058 (0x3059 bytes) 000394[0] 1779068937.519531 mapwrite 0x058000 thru 0x067fcb (0xffcc bytes) 000395[1] 1779068937.640893 trunc from 0x1ead4f to 0x85d0e4 (0x672396 bytes) 000396[1] 1779068937.673892 trunc from 0x85d0e5 to 0x282964 (0x5da780 bytes) 000397[1] 1779068937.687671 read_OD 0x1a1000 thru 0x1a3fff (0x3000 bytes) 000400[1] 1779068937.704058 trunc from 0x282965 to 0x094a0a (0x1edf5a bytes) 000402[1] 1779068937.728782 read_OD 0x08f000 thru 0x093fff (0x5000 bytes) 000404[1] 1779068937.745751 mapwrite 0x70f000 thru 0x711ff4 (0x2ff5 bytes) 000405[0] 1779068937.985588 mapread 0x45c000 thru 0x46b455 (0xf456 bytes) 000406[1] 1779068938.059919 trunc from 0x711ff5 to 0x17bccf (0x596325 bytes) 000410[1] 1779068938.140402 mapread 0x019000 thru 0x028bc8 (0xfbc9 bytes) 000412[1] 1779068938.230839 mapwrite 0x175000 thru 0x17aebe (0x5ebf bytes) 000415[1] 1779068938.386024 mapread 0x15d000 thru 0x168a99 (0xba9a bytes) 000416[1] 1779068938.462540 read_OD 0x0f0000 thru 0x0f8fff (0x9000 bytes) 000422[0] 1779068938.495933 write_OD 0x891000 thru 0x899fff (0x9000 bytes) 000423[1] 1779068938.616911 mapread 0x02d000 thru 0x037747 (0xa748 bytes) 000426[0] 1779068938.714582 trunc from 0x89a000 to 0x37f7ea (0x51a815 bytes) 000427[1] 1779068938.775966 mapwrite 0x454000 thru 0x45b455 (0x7456 bytes) 000428[0] 1779068938.859502 read_OD 0x286000 thru 0x28efff (0x9000 bytes) 000429[1] 1779068938.884619 mapread 0x3d3000 thru 0x3d4bef (0x1bf0 bytes) 000430[0] 1779068938.895867 read_OD 0x43b000 thru 0x448fff (0xe000 bytes) 000434[0] 1779068938.904301 trunc from 0x45b456 to 0x43580b (0x25c4a bytes) 000435[1] 1779068938.939871 mapread 0x315000 thru 0x31d267 (0x8268 bytes) 000436[1] 1779068939.032950 mapwrite 0x2a7000 thru 0x2b12a9 (0xa2aa bytes) 000438[0] 1779068939.164031 mapwrite 0x4ce000 thru 0x4d1499 (0x349a bytes) 000440[1] 1779068939.268750 write_OD 0x17b000 thru 0x187fff (0xd000 bytes) 000441[1] 1779068939.340946 read_OD 0x398000 thru 0x39bfff (0x4000 bytes) 000448[0] 1779068939.353576 mapwrite 0x9b2000 thru 0x9c07a9 (0xe7aa bytes) 000449[0] 1779068939.474007 read_OD 0x205000 thru 0x20dfff (0x9000 bytes) 000452[0] 1779068939.499017 trunc from 0x9c07aa to 0x468519 (0x558290 bytes) 000455[1] 1779068939.527966 write_OD 0x848000 thru 0x84bfff (0x4000 bytes) 000456[1] 1779068939.589770 mapread 0x5b0000 thru 0x5b0faa (0xfab bytes) 000458[1] 1779068939.596042 mapread 0x097000 thru 0x09d4c4 (0x64c5 bytes) 000459[0] 1779068939.615174 mapread 0x7d9000 thru 0x7dcfdf (0x3fe0 bytes) 000460[1] 1779068939.631165 read_OD 0x6fb000 thru 0x702fff (0x8000 bytes) 000462[1] 1779068939.639138 write_OD 0x841000 thru 0x84bfff (0xb000 bytes) 000463[1] 1779068939.686181 write_OD 0x816000 thru 0x81dfff (0x8000 bytes) 000465[1] 1779068939.731981 read_OD 0x6d7000 thru 0x6dcfff (0x6000 bytes) 000468[0] 1779068939.740108 trunc from 0x84c000 to 0x8d7441 (0x8b442 bytes) 000469[0] 1779068939.794254 write_OD 0x3af000 thru 0x3b9fff (0xb000 bytes) 000470[1] 1779068939.844071 read_OD 0x103000 thru 0x10cfff (0xa000 bytes) 000476[0] 1779068939.855869 write_OD 0x91d000 thru 0x923fff (0x7000 bytes) 000478[0] 1779068939.909283 read_OD 0x62c000 thru 0x638fff (0xd000 bytes) 000481[1] 1779068939.917538 write_OD 0x01f000 thru 0x02bfff (0xd000 bytes) 000482[0] 1779068939.964290 read_OD 0x02d000 thru 0x02efff (0x2000 bytes) 000483[1] 1779068939.973976 read_OD 0x35f000 thru 0x369fff (0xb000 bytes) 000484[0] 1779068939.985769 read_OD 0x91f000 thru 0x920fff (0x2000 bytes) 000487[0] 1779068939.992031 write_OD 0x1c1000 thru 0x1c7fff (0x7000 bytes) 000488[0] 1779068940.041800 write_OD 0x427000 thru 0x42bfff (0x5000 bytes) 000489[1] 1779068940.070727 write_OD 0x78b000 thru 0x796fff (0xc000 bytes) 000490[0] 1779068940.097882 mapread 0x443000 thru 0x44d135 (0xa136 bytes) 000491[0] 1779068940.118934 read_OD 0x13e000 thru 0x148fff (0xb000 bytes) 000494[0] 1779068940.136886 write_OD 0x6a4000 thru 0x6b1fff (0xe000 bytes) 000495[0] 1779068940.195739 mapwrite 0x542000 thru 0x545cbc (0x3cbd bytes) 000497[0] 1779068940.326922 read_OD 0x6d1000 thru 0x6dffff (0xf000 bytes) 000499[0] 1779068940.340183 trunc from 0x924000 to 0x316ce0 (0x60d31f bytes) 000500[0] 1779068940.387423 read_OD 0x21b000 thru 0x224fff (0xa000 bytes) All operations completed A-OK! PASS 16a (46s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 21:49:04 (1779068944) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.352634 s, 23.8 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.287558 s, 29.2 MB/s Chance of close/open is 1 in 50 Seed set to 8946 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read skipping zero size read skipping zero size read skipping zero size read skipping zero size read skipping zero size read 000014[1] 1779068947.379274 mapwrite 0x35231a thru 0x352f7b (0xc62 bytes) truncating to largest ever: 0x636c0 000015[0] 1779068947.596942 trunc from 0x352f7c to 0x0636bf (0x2ef8bc bytes) 000018[1] 1779068947.721760 read 0x062bfa thru 0x0636bf (0xac6 bytes) 000021[0] 1779068947.741717 mapwrite 0x4cefed thru 0x4cfa5f (0xa73 bytes) 000024[1] 1779068947.850606 read 0x3a9d6b thru 0x3aa2f4 (0x58a bytes) 000029[1] 1779068947.923806 read 0x014426 thru 0x015a4a (0x1625 bytes) 000030[1] 1779068947.927944 mapread 0x2a708f thru 0x2a835b (0x12cd bytes) truncating to largest ever: 0x103c68 000031[0] 1779068947.966063 trunc from 0x4cfa60 to 0x103c67 (0x3cbdf8 bytes) truncating to largest ever: 0x1b4a94 000032[0] 1779068948.042268 trunc from 0x103c68 to 0x1b4a93 (0xb0e2c bytes) 000034[0] 1779068948.091903 write 0x9f839c thru 0x9f9348 (0xfad bytes) truncating to largest ever: 0x8c7017 000035[0] 1779068948.094865 trunc from 0x9f9349 to 0x8c7016 (0x132332 bytes) 000039[0] 1779068948.191941 read 0x8442c3 thru 0x845955 (0x1693 bytes) 000041[0] 1779068948.202879 write 0x359e66 thru 0x35a44e (0x5e9 bytes) 000044[1] 1779068948.252752 trunc from 0x8c7017 to 0x108b1d (0x7be4f9 bytes) 000045[1] 1779068948.317620 trunc from 0x108b1e to 0x09511b (0x73a02 bytes) 000046[1] 1779068948.349204 mapwrite 0x19b474 thru 0x19bcbb (0x848 bytes) 000047[0] 1779068948.439510 mapread 0x1623c1 thru 0x163133 (0xd73 bytes) 000049[1] 1779068948.495926 mapread 0x114dee thru 0x116b1b (0x1d2e bytes) 000050[0] 1779068948.523646 read 0x11ba3e thru 0x11cc7f (0x1242 bytes) 000052[1] 1779068948.587951 read 0x09c3bc thru 0x09d5eb (0x1230 bytes) 000059[1] 1779068948.671974 mapread 0x13641a thru 0x13643b (0x22 bytes) 000061[1] 1779068948.687461 read 0x02b4c4 thru 0x02cf91 (0x1ace bytes) 000063[0] 1779068948.703299 mapread 0x0f0872 thru 0x0f20a7 (0x1836 bytes) 000066[1] 1779068948.712309 mapread 0x0a9fdb thru 0x0aa375 (0x39b bytes) 000069[1] 1779068948.740089 mapread 0x1177e6 thru 0x118bc1 (0x13dc bytes) 000070[1] 1779068948.741178 mapread 0x02ed44 thru 0x02fac2 (0xd7f bytes) 000073[0] 1779068948.791840 read 0x14deb5 thru 0x14fb14 (0x1c60 bytes) truncating to largest ever: 0x9aae49 000082[0] 1779068948.859857 trunc from 0x19bcbc to 0x9aae48 (0x80f18d bytes) 000083[0] 1779068948.945721 trunc from 0x9aae49 to 0x39a3cb (0x610a7d bytes) 000084[0] 1779068948.989997 write 0x00135b thru 0x0026de (0x1384 bytes) 000088[0] 1779068949.157828 read 0x339cbb thru 0x33b285 (0x15cb bytes) 000093[0] 1779068949.244732 trunc from 0x39a3cc to 0x801a8d (0x4676c2 bytes) 000095[1] 1779068949.269318 mapwrite 0x7ab6cb thru 0x7ac6d8 (0x100e bytes) 000096[0] 1779068949.374317 read 0x579c2d thru 0x57ab95 (0xf69 bytes) 000098[0] 1779068949.419183 mapwrite 0x6356ae thru 0x6374af (0x1e02 bytes) 000099[0] 1779068949.491083 read 0x30fa6e thru 0x31056d (0xb00 bytes) 000100[1] 1779068949.516527 trunc from 0x801a8e to 0x5a5061 (0x25ca2c bytes) 000101[0] 1779068949.577267 trunc from 0x5a5062 to 0x3dc204 (0x1c8e5d bytes) 000102[0] 1779068949.628313 mapread 0x2af5a0 thru 0x2b054e (0xfaf bytes) 000108[0] 1779068949.735182 trunc from 0x3dc205 to 0x3650ae (0x77156 bytes) 000110[0] 1779068949.792980 trunc from 0x3650af to 0x3c009a (0x5afec bytes) 000111[0] 1779068949.836726 mapread 0x110f13 thru 0x111aee (0xbdc bytes) 000112[0] 1779068949.877446 mapwrite 0x19a175 thru 0x19bb24 (0x19b0 bytes) 000119[0] 1779068950.084661 mapwrite 0x94e0da thru 0x94f868 (0x178f bytes) 000121[0] 1779068950.195040 mapwrite 0x6b3c88 thru 0x6b49af (0xd28 bytes) 000123[0] 1779068950.293292 trunc from 0x94f869 to 0x88af19 (0xc494f bytes) 000124[1] 1779068950.312818 trunc from 0x88af1a to 0x38075a (0x50a7bf bytes) 000127[1] 1779068950.376721 trunc from 0x38075b to 0x96ab3e (0x5ea3e4 bytes) 000129[1] 1779068950.426297 read 0x9359fc thru 0x937922 (0x1f27 bytes) 000130[1] 1779068950.453773 mapwrite 0x433907 thru 0x434cf4 (0x13ee bytes) 000131[1] 1779068950.510714 trunc from 0x96ab3f to 0x2c7b4e (0x6a2ff0 bytes) 000132[0] 1779068950.545792 trunc from 0x2c7b4f to 0x2a6ad7 (0x21077 bytes) 000135[1] 1779068950.615294 trunc from 0x2a6ad8 to 0x1eb564 (0xbb573 bytes) 000136[0] 1779068950.682142 trunc from 0x1eb565 to 0x4b74c2 (0x2cbf5e bytes) 000137[1] 1779068950.737086 trunc from 0x4b74c3 to 0x09a102 (0x41d3c0 bytes) 000140[1] 1779068950.847707 write 0x5a4cd9 thru 0x5a5b01 (0xe29 bytes) 000141[0] 1779068950.872893 read 0x569879 thru 0x56b482 (0x1c0a bytes) 000143[0] 1779068950.926303 trunc from 0x5a5b02 to 0x84d48b (0x2a798a bytes) 000144[1] 1779068950.993060 trunc from 0x84d48c to 0x525c82 (0x327809 bytes) 000147[0] 1779068951.061799 mapread 0x138d8d thru 0x139707 (0x97b bytes) 000148[0] 1779068951.100477 read 0x4af79e thru 0x4b152d (0x1d90 bytes) 000152[1] 1779068951.132512 mapwrite 0x6a2116 thru 0x6a282f (0x71a bytes) 000153[1] 1779068951.262238 mapread 0x0955c2 thru 0x095879 (0x2b8 bytes) 000160[0] 1779068951.331900 write 0x0e8124 thru 0x0e9e3a (0x1d17 bytes) 000162[1] 1779068951.416305 trunc from 0x6a2830 to 0x6a48f7 (0x20c8 bytes) 000166[1] 1779068951.552063 mapwrite 0x11bab4 thru 0x11c899 (0xde6 bytes) 000171[0] 1779068951.734036 write 0x103562 thru 0x103ea5 (0x944 bytes) 000173[1] 1779068951.821734 write 0x080591 thru 0x08223f (0x1caf bytes) 000175[1] 1779068951.926766 trunc from 0x6a48f8 to 0x76990a (0xc5013 bytes) 000177[1] 1779068951.999207 mapread 0x3dc1d8 thru 0x3dc2f9 (0x122 bytes) 000178[0] 1779068952.012157 write 0x085f3f thru 0x087efa (0x1fbc bytes) 000184[1] 1779068952.163862 mapwrite 0x0b0bb4 thru 0x0b1c81 (0x10ce bytes) truncating to largest ever: 0x9ff561 000185[0] 1779068952.289914 trunc from 0x76990b to 0x9ff560 (0x295c56 bytes) 000186[0] 1779068952.351294 read 0x640732 thru 0x641f1c (0x17eb bytes) 000188[0] 1779068952.384194 trunc from 0x9ff561 to 0x0c5378 (0x93a1e8 bytes) 000189[0] 1779068952.453471 mapread 0x01fc15 thru 0x02076d (0xb59 bytes) 000190[0] 1779068952.517196 mapread 0x0b24c6 thru 0x0b25ad (0xe8 bytes) 000193[1] 1779068952.611943 trunc from 0x0c5379 to 0x9a49d1 (0x8df659 bytes) 000194[0] 1779068952.687658 read 0x0d0e72 thru 0x0d2c76 (0x1e05 bytes) 000199[0] 1779068952.900839 read 0x3f4566 thru 0x3f45d4 (0x6f bytes) 000200[0] 1779068952.917364 read 0x09a90a thru 0x09c5c6 (0x1cbd bytes) 000202[1] 1779068952.936075 write 0x32aaf2 thru 0x32bdc7 (0x12d6 bytes) 000204[0] 1779068953.013617 read 0x573af7 thru 0x5753de (0x18e8 bytes) 000205[1] 1779068953.037976 trunc from 0x9a49d2 to 0x7d9a4d (0x1caf84 bytes) 000206[0] 1779068953.084229 trunc from 0x7d9a4e to 0x3ab50e (0x42e53f bytes) 000207[1] 1779068953.146091 read 0x1dffe6 thru 0x1e1585 (0x15a0 bytes) 000209[0] 1779068953.254718 read 0x0bcaff thru 0x0be476 (0x1978 bytes) 000210[0] 1779068953.371908 mapwrite 0x01f5b8 thru 0x01fda5 (0x7ee bytes) 000211[1] 1779068953.452707 read 0x11d783 thru 0x11f1b0 (0x1a2e bytes) 000214[1] 1779068953.568700 trunc from 0x3ab50f to 0x016d8e (0x394780 bytes) 000215[0] 1779068953.670864 mapread 0x009098 thru 0x00a944 (0x18ad bytes) 000216[1] 1779068953.720028 mapread 0x01557a thru 0x015c20 (0x6a7 bytes) 000219[1] 1779068953.744090 write 0x2dbeda thru 0x2ddc11 (0x1d38 bytes) 000221[1] 1779068953.808319 trunc from 0x2ddc12 to 0x79ea4a (0x4c0e39 bytes) 000222[0] 1779068953.839141 read 0x1ed3d3 thru 0x1eebb7 (0x17e5 bytes) 000223[1] 1779068953.952748 write 0x0286a8 thru 0x02a1ee (0x1b47 bytes) 000224[1] 1779068953.982356 mapwrite 0x1004fd thru 0x100f44 (0xa48 bytes) 000226[1] 1779068954.047800 mapread 0x033263 thru 0x033e44 (0xbe2 bytes) 000228[1] 1779068954.102063 trunc from 0x79ea4b to 0x53b260 (0x2637ea bytes) 000229[0] 1779068954.124610 mapwrite 0x55a3a7 thru 0x55b698 (0x12f2 bytes) 000232[1] 1779068954.305898 mapread 0x3fb676 thru 0x3fbc39 (0x5c4 bytes) 000234[1] 1779068954.334679 read 0x22fbc4 thru 0x230c21 (0x105e bytes) 000238[0] 1779068954.384084 write 0x44f7af thru 0x450573 (0xdc5 bytes) 000239[1] 1779068954.385788 trunc from 0x55b699 to 0x4019bc (0x159cdc bytes) 000240[0] 1779068954.460855 write 0x8bc48a thru 0x8bdecd (0x1a44 bytes) 000242[0] 1779068954.507938 mapread 0x28ec4b thru 0x28ed5a (0x110 bytes) 000244[0] 1779068954.567314 mapwrite 0x2c0c02 thru 0x2c1871 (0xc70 bytes) 000245[0] 1779068954.636004 trunc from 0x8bdece to 0x3a9db1 (0x51411c bytes) 000247[1] 1779068954.711794 write 0x22748b thru 0x227a06 (0x57c bytes) 000248[0] 1779068954.732896 read 0x1af016 thru 0x1b0ed8 (0x1ec3 bytes) 000250[1] 1779068954.839656 read 0x332de7 thru 0x3344bd (0x16d7 bytes) 000251[0] 1779068954.904446 trunc from 0x3a9db2 to 0x38366b (0x26746 bytes) 000253[1] 1779068954.965785 mapwrite 0x52ffac thru 0x5305f0 (0x645 bytes) 000254[1] 1779068955.094978 mapread 0x47fa94 thru 0x480d57 (0x12c4 bytes) 000255[1] 1779068955.132048 read 0x4c7bd0 thru 0x4c8e45 (0x1276 bytes) 000259[1] 1779068955.175768 write 0x6d4359 thru 0x6d5245 (0xeed bytes) 000260[1] 1779068955.187081 mapread 0x41ea3b thru 0x420974 (0x1f3a bytes) 000261[0] 1779068955.199528 trunc from 0x6d5246 to 0x7cbd61 (0xf6b1c bytes) 000263[1] 1779068955.304259 trunc from 0x7cbd62 to 0x63fb45 (0x18c21c bytes) 000264[0] 1779068955.368727 mapread 0x2aee36 thru 0x2aff3c (0x1107 bytes) 000265[1] 1779068955.442810 write 0x69a28a thru 0x69be6a (0x1be1 bytes) 000266[0] 1779068955.452787 read 0x595a03 thru 0x5962b7 (0x8b5 bytes) 000268[1] 1779068955.493936 write 0x14226b thru 0x1427d2 (0x568 bytes) 000269[0] 1779068955.551845 mapwrite 0x11fc28 thru 0x1202ee (0x6c7 bytes) 000270[1] 1779068955.682083 read 0x04ad14 thru 0x04c0b5 (0x13a2 bytes) 000271[0] 1779068955.711660 write 0x2d3519 thru 0x2d35b1 (0x99 bytes) 000272[0] 1779068955.761792 read 0x03089b thru 0x030a29 (0x18f bytes) 000273[0] 1779068955.767582 mapread 0x11a8cf thru 0x11c48a (0x1bbc bytes) 000281[1] 1779068955.841040 mapwrite 0x4ae2f1 thru 0x4afd3b (0x1a4b bytes) 000282[0] 1779068955.987909 mapread 0x0a61f7 thru 0x0a7857 (0x1661 bytes) 000283[0] 1779068956.020868 mapread 0x1324b6 thru 0x132e13 (0x95e bytes) 000284[0] 1779068956.069083 write 0x2dacba thru 0x2db3a5 (0x6ec bytes) 000285[1] 1779068956.098242 mapwrite 0x66fb09 thru 0x67086e (0xd66 bytes) 000286[0] 1779068956.194986 write 0x6b1f45 thru 0x6b39b5 (0x1a71 bytes) 000291[1] 1779068956.265851 trunc from 0x6b39b6 to 0x7c944c (0x115a97 bytes) 000293[0] 1779068956.308273 mapwrite 0x3a52b4 thru 0x3a70aa (0x1df7 bytes) 000297[1] 1779068956.505122 write 0x41920c thru 0x419c7d (0xa72 bytes) 000299[1] 1779068956.547786 write 0x667281 thru 0x6686b0 (0x1430 bytes) 000304[0] 1779068956.667039 trunc from 0x7c944d to 0x3e22b7 (0x3e7195 bytes) 000305[1] 1779068956.728463 trunc from 0x3e22b8 to 0x505c90 (0x1239d9 bytes) 000306[1] 1779068956.814306 read 0x0a29b7 thru 0x0a3822 (0xe6c bytes) 000309[0] 1779068956.845793 mapwrite 0x39b1cf thru 0x39c9ea (0x181c bytes) 000312[0] 1779068956.967544 mapwrite 0x16e811 thru 0x16ef48 (0x738 bytes) 000313[1] 1779068957.010528 mapwrite 0x4e08f1 thru 0x4e15fa (0xd0a bytes) 000314[1] 1779068957.064667 write 0x1449fe thru 0x1467d9 (0x1ddc bytes) 000315[0] 1779068957.075768 trunc from 0x505c91 to 0x00337d (0x502913 bytes) 000316[1] 1779068957.137720 mapread 0x0026c6 thru 0x002c42 (0x57d bytes) 000318[1] 1779068957.178586 mapread 0x002e84 thru 0x00337d (0x4fa bytes) 000324[1] 1779068957.236862 write 0x0428b1 thru 0x043d1a (0x146a bytes) 000326[1] 1779068957.277107 write 0x50815b thru 0x508d21 (0xbc7 bytes) 000333[1] 1779068957.335923 mapread 0x04a381 thru 0x04b5e8 (0x1268 bytes) 000334[1] 1779068957.342870 write 0x0ecb45 thru 0x0ed894 (0xd50 bytes) 000336[0] 1779068957.373609 trunc from 0x508d22 to 0x3331f7 (0x1d5b2a bytes) 000337[0] 1779068957.437685 read 0x1c98d6 thru 0x1cb275 (0x19a0 bytes) 000339[1] 1779068957.530375 mapread 0x049186 thru 0x04aea2 (0x1d1d bytes) 000340[1] 1779068957.576224 write 0x4b5024 thru 0x4b51c7 (0x1a4 bytes) 000341[0] 1779068957.615113 write 0x31298b thru 0x31338b (0xa01 bytes) 000343[0] 1779068957.668274 mapread 0x1b8ba0 thru 0x1baa5c (0x1ebd bytes) 000344[0] 1779068957.739684 read 0x447896 thru 0x449278 (0x19e3 bytes) 000346[1] 1779068957.786243 read 0x0a5204 thru 0x0a6710 (0x150d bytes) 000348[1] 1779068957.813025 trunc from 0x4b51c8 to 0x8fe404 (0x44923d bytes) 000349[1] 1779068957.859385 write 0x885163 thru 0x88655e (0x13fc bytes) 000350[0] 1779068957.869032 write 0x39b251 thru 0x39cb83 (0x1933 bytes) 000352[1] 1779068957.919112 mapwrite 0x949627 thru 0x94a285 (0xc5f bytes) 000353[1] 1779068958.004613 write 0x157a54 thru 0x1584d2 (0xa7f bytes) 000355[1] 1779068958.063268 read 0x045b78 thru 0x046a5b (0xee4 bytes) 000356[0] 1779068958.067995 mapread 0x6660c4 thru 0x666393 (0x2d0 bytes) 000358[0] 1779068958.114080 mapread 0x515c3c thru 0x5174b1 (0x1876 bytes) 000360[1] 1779068958.149393 read 0x440994 thru 0x442290 (0x18fd bytes) 000361[0] 1779068958.159026 read 0x059204 thru 0x059c3e (0xa3b bytes) 000362[1] 1779068958.183793 write 0x5488e3 thru 0x54a17d (0x189b bytes) 000364[0] 1779068958.229817 read 0x167768 thru 0x169065 (0x18fe bytes) 000366[1] 1779068958.252404 read 0x057af9 thru 0x059313 (0x181b bytes) 000367[1] 1779068958.271897 mapwrite 0x940d6a thru 0x9415a1 (0x838 bytes) 000370[1] 1779068958.361821 mapwrite 0x929eaf thru 0x92a259 (0x3ab bytes) 000372[0] 1779068958.421447 mapread 0x4dc0a9 thru 0x4ddb8e (0x1ae6 bytes) 000374[1] 1779068958.459021 write 0x6aee7a thru 0x6b0b09 (0x1c90 bytes) 000376[0] 1779068958.480870 mapwrite 0x1c7f66 thru 0x1c9a8e (0x1b29 bytes) 000377[1] 1779068958.578715 write 0x6957cf thru 0x697278 (0x1aaa bytes) 000379[0] 1779068958.609021 mapread 0x0eb905 thru 0x0ed76d (0x1e69 bytes) 000380[1] 1779068958.633843 write 0x72068a thru 0x72104e (0x9c5 bytes) 000384[0] 1779068958.697220 mapread 0x216879 thru 0x2170e3 (0x86b bytes) 000385[1] 1779068958.731675 mapwrite 0x3bac83 thru 0x3bc87a (0x1bf8 bytes) 000387[1] 1779068958.838662 mapwrite 0x1f8727 thru 0x1f8cdf (0x5b9 bytes) 000388[1] 1779068958.878267 write 0x884108 thru 0x885efd (0x1df6 bytes) 000389[0] 1779068958.886231 trunc from 0x94a286 to 0x9878d5 (0x3d650 bytes) 000391[1] 1779068958.942649 write 0x92df1a thru 0x92df9d (0x84 bytes) 000394[0] 1779068959.027885 mapread 0x5c3d3a thru 0x5c540e (0x16d5 bytes) 000399[1] 1779068959.223287 mapread 0x698645 thru 0x69a4a3 (0x1e5f bytes) 000402[1] 1779068959.349862 mapread 0x62d14b thru 0x62f088 (0x1f3e bytes) 000404[0] 1779068959.390941 trunc from 0x9878d6 to 0x71f918 (0x267fbd bytes) 000405[0] 1779068959.461568 read 0x205c30 thru 0x206287 (0x658 bytes) 000406[0] 1779068959.470744 trunc from 0x71f919 to 0x1f3015 (0x52c903 bytes) 000407[0] 1779068959.515064 read 0x1ab45d thru 0x1abd39 (0x8dd bytes) 000408[0] 1779068959.572728 mapread 0x1010b0 thru 0x102b13 (0x1a64 bytes) 000409[1] 1779068959.598183 read 0x041c4c thru 0x0427db (0xb90 bytes) 000410[0] 1779068959.698030 trunc from 0x1f3016 to 0x9b4004 (0x7c0fef bytes) 000412[1] 1779068959.822870 mapwrite 0x3b8c73 thru 0x3baa60 (0x1dee bytes) 000415[1] 1779068960.030151 write 0x422e65 thru 0x423be5 (0xd81 bytes) 000418[0] 1779068960.144722 read 0x12f77c thru 0x1303a1 (0xc26 bytes) 000419[1] 1779068960.171168 mapwrite 0x8692e5 thru 0x8694c1 (0x1dd bytes) 000420[1] 1779068960.270018 mapread 0x05b1bd thru 0x05b63e (0x482 bytes) 000422[0] 1779068960.302037 write 0x57c20f thru 0x57d787 (0x1579 bytes) 000423[0] 1779068960.361198 write 0x0ea45a thru 0x0ebb31 (0x16d8 bytes) 000425[1] 1779068960.406237 mapwrite 0x9f2aaf thru 0x9f4a7b (0x1fcd bytes) 000427[1] 1779068960.513896 trunc from 0x9f4a7c to 0x8c696a (0x12e111 bytes) 000429[1] 1779068960.553820 mapwrite 0x6a99c2 thru 0x6ab73b (0x1d7a bytes) 000431[1] 1779068960.646733 mapwrite 0x7d49f0 thru 0x7d5211 (0x822 bytes) 000435[0] 1779068960.784892 read 0x613b9c thru 0x614b34 (0xf99 bytes) 000436[1] 1779068960.803097 mapread 0x4cdff2 thru 0x4ceb70 (0xb7f bytes) 000437[1] 1779068960.831100 read 0x0fea4d thru 0x0ff1f6 (0x7aa bytes) 000438[0] 1779068960.850832 trunc from 0x8c696b to 0x3aee8a (0x517ae0 bytes) 000439[1] 1779068960.899703 read 0x22f3f2 thru 0x230653 (0x1262 bytes) 000440[0] 1779068960.918369 mapwrite 0x1b57e1 thru 0x1b7687 (0x1ea7 bytes) 000441[0] 1779068960.990608 mapwrite 0x30314f thru 0x303f57 (0xe09 bytes) 000442[0] 1779068961.037795 write 0x2fc154 thru 0x2fdba2 (0x1a4f bytes) 000447[1] 1779068961.130469 read 0x229c3c thru 0x22b744 (0x1b09 bytes) 000448[0] 1779068961.228824 write 0x7ee7d1 thru 0x7f037d (0x1bad bytes) 000450[1] 1779068961.231768 mapread 0x226740 thru 0x226e9d (0x75e bytes) 000452[1] 1779068961.249979 mapread 0x470a6a thru 0x471640 (0xbd7 bytes) 000458[0] 1779068961.281268 mapwrite 0x193541 thru 0x1941fa (0xcba bytes) 000460[0] 1779068961.388778 trunc from 0x7f037e to 0x53bd83 (0x2b45fa bytes) 000462[1] 1779068961.456860 read 0x24c98f thru 0x24dea3 (0x1515 bytes) 000463[0] 1779068961.484727 write 0x0d70c8 thru 0x0d87df (0x1718 bytes) 000465[0] 1779068961.553902 mapread 0x4453c3 thru 0x445edc (0xb1a bytes) 000466[0] 1779068961.565197 mapwrite 0x374b3f thru 0x3759a8 (0xe6a bytes) 000467[0] 1779068961.705467 write 0x549d73 thru 0x54b40c (0x169a bytes) 000468[0] 1779068961.711399 mapread 0x01e5f4 thru 0x0205ef (0x1ffc bytes) 000470[1] 1779068961.775107 read 0x041445 thru 0x042e89 (0x1a45 bytes) 000473[0] 1779068961.830744 mapread 0x0547fb thru 0x055be4 (0x13ea bytes) 000474[0] 1779068961.852798 mapwrite 0x8d397f thru 0x8d4756 (0xdd8 bytes) 000475[0] 1779068961.950468 mapread 0x381ffe thru 0x382919 (0x91c bytes) 000476[0] 1779068962.004084 read 0x33a920 thru 0x33ae64 (0x545 bytes) 000485[1] 1779068962.084236 mapread 0x815c74 thru 0x81625f (0x5ec bytes) 000488[1] 1779068962.121221 write 0x7fbff2 thru 0x7fd322 (0x1331 bytes) 000491[1] 1779068962.168200 mapread 0x298545 thru 0x29982f (0x12eb bytes) 000492[0] 1779068962.200723 mapwrite 0x066c56 thru 0x0688e5 (0x1c90 bytes) 000496[1] 1779068962.377470 write 0x5e2fc3 thru 0x5e3339 (0x377 bytes) 000500[1] 1779068962.447795 mapread 0x8771b7 thru 0x879012 (0x1e5c bytes) All operations completed A-OK! PASS 16b (23s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 21:49:27 (1779068967) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (3s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 21:49:30 (1779068970) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.305882 s, 27.4 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.225777 s, 37.2 MB/s ldlm.namespaces.MGC192.168.202.130@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff923242ae4000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff92325195d800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff923242ae4000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff92325195d800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff923242ae4000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff92325195d800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.56128 s, 12.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.67528 s, 39.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.46227 s, 42.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.19899 s, 20.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.16111 s, 12.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.97127 s, 35.3 MB/s PASS 16d (41s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 21:50:11 (1779069011) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.250047 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101983 s, 10.3 MB/s PASS 16e (5s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 21:50:16 (1779069016) TIMEOUT PASS 16f (28s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 21:50:44 (1779069044) TIMEOUT PASS 16g (27s) == sanityn test 16h: mmap read after truncate file ======= 21:51:11 (1779069071) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (7s) == sanityn test 16i: read after truncate file ============ 21:51:18 (1779069078) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (6s) == sanityn test 16j: race dio with buffered i/o ========== 21:51:24 (1779069084) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.50884 s, 17.4 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.16397 s, 22.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.24947 s, 11.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.39974 s, 18.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.01195 s, 13.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.7013 s, 15.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.07552 s, 12.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.87595 s, 14.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.91355 s, 13.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.63947 s, 9.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.7568 s, 9.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.05586 s, 12.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.13736 s, 12.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.58183 s, 10.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.64 s, 9.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.0349 s, 12.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.25848 s, 11.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.70957 s, 9.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.81337 s, 9.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (32s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 21:51:56 (1779069116) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported truncating to largest ever: 0x232e1d3 fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 All operations completed A-OK! Sun May 17 21:52:28 EDT 2026 + fsx 2 done: rc=0 Sun May 17 21:52:28 EDT 2026 All operations completed A-OK! All operations completed A-OK! + fsx 3 done: rc=0 Sun May 17 21:52:28 EDT 2026 + fsx 4 done: rc=0 Sun May 17 21:52:28 EDT 2026 + fsx 5 done: rc=0 Sun May 17 21:52:28 EDT 2026 PASS 16k (39s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 21:52:36 (1779069156) fail_loc=0x8000030a PASS 17 (9s) == sanityn test 18: mmap sanity check =========================================================================================== 21:52:45 (1779069165) mmap test1: basic mmap operation (PASS, 0.11074s) mmap test2: MAP_PRIVATE not write back (PASS, 0.061246s) mmap test3: concurrent mmap ops on two nodes (PASS, 10.109s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.5073s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.12944s) mmap test6: check mmap write/read content on two nodes (PASS, 0.17623s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 0.91438s) mmap test11: multi-thread shared mmap (PASS, 0.97846s) mmap test10: mtime not change for readonly mmap access (PASS, 1.085s) PASS 18 (24s) == sanityn test 19: test concurrent uncached read races ========================================================================= 21:53:09 (1779069189) oleg230-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg230-client: oleg230-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (4s) == sanityn test 20: test extra readahead page left in cache ============================================================== 21:53:13 (1779069193) PASS 20 (5s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 21:53:18 (1779069198) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (5s) == sanityn test 23: others should see updated atime while another read============================================================== 21:53:23 (1779069203) now is 1779069205 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7571 new atime is 1779069266 PASS 23 (67s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 21:54:30 (1779069270) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.6K 388 490.2K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.2K 399 114.8K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 401 114.2K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 388 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.6K 388 490.2K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 399 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 401 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 388 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 3.2M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 8.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 30.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502379 388 501991 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 117999 399 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117297 401 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234884 388 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.6K 388 490.2K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 399 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 401 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 388 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502379 388 501991 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 117999 399 117600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117297 401 116896 1% /mnt/lustre[OST:1] filesystem_summary: 234884 388 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502379 388 501991 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 117999 399 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117297 401 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234884 388 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (6s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 21:54:36 (1779069276) PASS 24b (6s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 21:54:42 (1779069282) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (5s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 21:54:47 (1779069287) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (2s) == sanityn test 26a: allow mtime to get older ============ 21:54:49 (1779069289) utime: good mknod times 1779069289 <= 1779069290 <= 1779069290 for /mnt/lustre/f26a utime: good mknod times 1779069289 <= 1779069290 <= 1779069290 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (6s) == sanityn test 26b: sync mtime between ost and mds ====== 21:54:55 (1779069295) PASS 26b (7s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 21:55:02 (1779069302) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7571 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-05-17 21:55:03.000000000 -0400 Birth: 2026-05-17 21:55:02.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-05-17 21:55:03.000000000 -0400 Birth: 2026-05-17 21:55:02.000000000 -0400 PASS 26c (6s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 21:55:09 (1779069309) 0 PASS 30 (7s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 21:55:16 (1779069316) fail_loc=0x314 PASS 31a (7s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 21:55:23 (1779069323) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg230-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.130@tcp:/lustre 7542784 40960 7497728 1% /mnt/lustre2 PASS 31b (17s) == sanityn test 31r: open-rename(replace) race =========== 21:55:40 (1779069340) -rw-r--r-- 1 root root 0 May 17 21:55 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (9s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 21:55:49 (1779069349) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1207: 42924 Terminated tail -f $tfile PASS 31s (6s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 21:55:55 (1779069355) PASS 31t (7s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 21:56:03 (1779069363) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (3s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 21:56:06 (1779069366) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (2s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 21:56:08 (1779069368) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (3s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 21:56:11 (1779069371) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (2s) == sanityn test 34: no lock timeout under IO ============= 21:56:13 (1779069373) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg230-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff923242ae4000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff923242ae4000.ost_server_uuid in IDLE state after 0 sec oleg230-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff923242ae4000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff923242ae4000.ost_server_uuid in IDLE state after 0 sec oleg230-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff923242ae4000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff923242ae4000.ost_server_uuid in IDLE state after 0 sec oleg230-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff923242ae4000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff923242ae4000.ost_server_uuid in IDLE state after 0 sec oleg230-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff923242ae4000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff923242ae4000.ost_server_uuid in IDLE state after 0 sec oleg230-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff923242ae4000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff923242ae4000.ost_server_uuid in IDLE state after 0 sec PASS 34 (116s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 21:58:09 (1779069489) fail_loc=0x80000317 Race attempt 0 Wait for 49046 49118 for 60 sec... - open/close 951 (time 1779069504.98 total 10.00 last 95.09) - open/close 1845 (time 1779069514.99 total 20.01 last 89.32) - open/close 2825 (time 1779069524.99 total 30.01 last 97.97) - open/close 3743 (time 1779069535.01 total 40.04 last 91.59) total: 4000 open/close in 42.53 seconds: 94.04 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.202.130@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre 192.168.202.130@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre2 PASS 35 (69s) == sanityn test 36: handle ESTALE/open-unlink correctly == 21:59:18 (1779069558) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg230-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.856401 s, 61.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7571 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7496704) after_dd(7438336) after(7489536) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.01228 s, 51.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7571 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7489536) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.700773 s, 74.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7571 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7492608) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.9481 s, 55.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7571 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7494656) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.939849 s, 55.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7571 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7492608) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.753649 s, 69.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7571 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7492608) after_dd(7439360) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.840563 s, 62.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7571 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7496704) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.751625 s, 69.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7571 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7494656) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.734101 s, 71.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7571 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7494656) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.65837 s, 79.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7571 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7494656) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.67826 s, 77.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7571 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7494656) after_dd(7445504) after(7496704) PASS 36 (264s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 22:03:42 (1779069822) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7571 - create 2181 (time 1779069834.94 total 10.00 last 218.09) - create 4638 (time 1779069844.94 total 20.00 last 245.69) - create 6746 (time 1779069854.94 total 30.00 last 210.75) - create 9130 (time 1779069864.94 total 40.00 last 238.37) total: 10000 create in 43.44 seconds: 230.21 ops/second PASS 37 (50s) == sanityn test 39a: file mtime does not change after rename ========================================================== 22:04:32 (1779069872) repeat after cancel_lru_locks PASS 39a (4s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 22:04:36 (1779069876) repeat after cancel_lru_locks PASS 39b (5s) == sanityn test 39c: check truncate mtime update ================================================================================ 22:04:41 (1779069881) repeat after cancel_lru_locks PASS 39c (5s) == sanityn test 39d: sync write should update mtime ====== 22:04:46 (1779069886) fail_loc=0x411 fail_loc=0 PASS 39d (3s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 22:04:50 (1779069890) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (12s) == sanityn test 40c: pdirops: link and others ======================================================================== 22:05:02 (1779069902) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (11s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 22:05:14 (1779069914) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (11s) == sanityn test 40e: pdirops: rename and others ======================================================================== 22:05:26 (1779069926) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (11s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 22:05:36 (1779069936) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (9s) == sanityn test 41b: pdirops: create vs create ======================================================================== 22:05:45 (1779069945) open: File exists Conflict PASS 41b (8s) == sanityn test 41c: pdirops: create vs link ======================================================================== 22:05:53 (1779069953) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (8s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 22:06:02 (1779069962) Conflict PASS 41d (8s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 22:06:10 (1779069970) Conflict PASS 41e (9s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 22:06:19 (1779069979) Conflict PASS 41f (8s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 22:06:27 (1779069987) Conflict PASS 41g (9s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 22:06:36 (1779069996) Conflict PASS 41h (9s) == sanityn test 41i: reint_open: create vs create ======== 22:06:45 (1779070005) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (802s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 22:20:07 (1779070807) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (6s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 22:20:13 (1779070813) open: File exists Conflict PASS 42b (5s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 22:20:18 (1779070818) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (6s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 22:20:24 (1779070824) Conflict PASS 42d (6s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 22:20:30 (1779070830) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (5s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 22:20:36 (1779070836) Conflict PASS 42f (6s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 22:20:42 (1779070842) Conflict PASS 42g (6s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 22:20:48 (1779070848) Conflict PASS 42h (6s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 22:20:54 (1779070854) PASS 43a (24s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 22:21:18 (1779070878) Conflict PASS 43b (6s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 22:21:24 (1779070884) Conflict PASS 43c (5s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 22:21:30 (1779070890) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (6s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 22:21:35 (1779070895) Conflict PASS 43e (6s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 22:21:41 (1779070901) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (5s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 22:21:46 (1779070906) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (5s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 22:21:51 (1779070911) Conflict PASS 43h (6s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 22:21:57 (1779070917) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (1s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 22:21:58 (1779070918) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists PASS 43j (52s) == sanityn test 43k: unlink vs create ==================== 22:22:50 (1779070970) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (478s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 22:30:48 (1779071448) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Sun May 17 22:30:52 EDT 2026 PASS 44a (5s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 22:30:54 (1779071454) open: File exists Conflict PASS 44b (6s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 22:30:59 (1779071459) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (5s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 22:31:04 (1779071464) Conflict PASS 44d (5s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 22:31:09 (1779071469) Conflict PASS 44e (6s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 22:31:15 (1779071475) Conflict PASS 44f (5s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 22:31:20 (1779071480) Conflict PASS 44g (5s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 22:31:25 (1779071485) Conflict PASS 44h (6s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 22:31:31 (1779071491) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (1s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 22:31:32 (1779071492) PASS 45a (29s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 22:32:01 (1779071521) Conflict PASS 45b (5s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 22:32:06 (1779071526) Conflict PASS 45c (6s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 22:32:12 (1779071532) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (5s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 22:32:17 (1779071537) Conflict PASS 45e (5s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 22:32:22 (1779071542) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (6s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 22:32:28 (1779071548) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (5s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 22:32:33 (1779071553) Conflict PASS 45h (5s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 22:32:38 (1779071558) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (1s) == sanityn test 45j: read vs rename ====================== 22:32:39 (1779071559) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (460s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 22:40:19 (1779072019) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (5s) == sanityn test 46b: pdirops: link vs create ======================================================================== 22:40:24 (1779072024) open: File exists Conflict PASS 46b (5s) == sanityn test 46c: pdirops: link vs link ======================================================================== 22:40:30 (1779072030) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (6s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 22:40:35 (1779072035) Conflict PASS 46d (5s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 22:40:40 (1779072040) Conflict PASS 46e (5s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 22:40:45 (1779072045) Conflict PASS 46f (5s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 22:40:50 (1779072050) Conflict PASS 46g (6s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 22:40:56 (1779072056) Conflict PASS 46h (5s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 22:41:01 (1779072061) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (1s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 22:41:02 (1779072062) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (1s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 22:41:03 (1779072063) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (1s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 22:41:04 (1779072064) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (1s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 22:41:05 (1779072065) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (1s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 22:41:06 (1779072066) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (1s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 22:41:07 (1779072067) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (1s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 22:41:08 (1779072068) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (1s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 22:41:09 (1779072069) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.00493747 s, 2.1 MB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (7s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 22:41:16 (1779072076) 0+1 records in 0+1 records out 158 bytes copied, 0.00307153 s, 51.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3510: kill: (259976) - No such process PASS 51a (4s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 22:41:20 (1779072080) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0027586 s, 371 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172838 s, 60.7 MB/s 3145728 PASS 51b (15s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 22:41:35 (1779072095) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0039997 s, 256 kB/s PASS 51c (6s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 22:41:41 (1779072101) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184966 s, 56.7 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (3s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 22:41:45 (1779072105) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2454 0x996 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2454 0x996 0x240000400 read lease(1) has applied. PASS 51e (5s) == sanityn test 54: rename locking ======================= 22:41:49 (1779072109) llite.lustre-ffff923242ae4000.xattr_cache=1 llite.lustre-ffff92325195d800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505011 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-17 22:41:49.000000000 -0400 Modify: 2026-05-17 22:41:49.000000000 -0400 Change: 2026-05-17 22:41:49.000000000 -0400 Birth: 2026-05-17 22:41:49.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505014 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-17 22:41:55.000000000 -0400 Modify: 2026-05-17 22:41:55.000000000 -0400 Change: 2026-05-17 22:41:55.000000000 -0400 Birth: 2026-05-17 22:41:54.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505017 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-17 22:42:00.000000000 -0400 Modify: 2026-05-17 22:42:00.000000000 -0400 Change: 2026-05-17 22:42:00.000000000 -0400 Birth: 2026-05-17 22:42:00.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505016 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-17 22:42:00.000000000 -0400 Modify: 2026-05-17 22:42:00.000000000 -0400 Change: 2026-05-17 22:42:00.000000000 -0400 Birth: 2026-05-17 22:42:00.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505019 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-17 22:42:06.000000000 -0400 Modify: 2026-05-17 22:42:06.000000000 -0400 Change: 2026-05-17 22:42:06.000000000 -0400 Birth: 2026-05-17 22:42:05.000000000 -0400 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-17 22:42:06.000000000 -0400 Modify: 2026-05-17 22:42:06.000000000 -0400 Change: 2026-05-17 22:42:06.000000000 -0400 Birth: 2026-05-17 22:42:05.000000000 -0400 PASS 54 (24s) == sanityn test 55a: rename vs unlink target dir ========= 22:42:13 (1779072133) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (7s) == sanityn test 55b: rename vs unlink source dir ========= 22:42:20 (1779072140) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (8s) == sanityn test 55c: rename vs unlink orphan target dir == 22:42:28 (1779072148) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (13s) == sanityn test 55d: rename file vs link ================= 22:42:41 (1779072161) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (9s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 22:42:50 (1779072170) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (2s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 22:42:52 (1779072172) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (13s) == sanityn test 55g: rename: race with trylock =========== 22:43:05 (1779072185) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (14s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 22:43:19 (1779072199) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1779072200 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 2.90662 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (24s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 22:43:43 (1779072223) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1779072225 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 3.59042 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (38s) == sanityn test 60: Verify data_version behaviour ======== 22:44:21 (1779072261) PASS 60 (2s) == sanityn test 70a: cd directory && rm directory ======== 22:44:23 (1779072263) PASS 70a (3s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 22:44:26 (1779072266) PASS 70b (3s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 22:44:29 (1779072269) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00256852 s, 25.5 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (1s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 22:44:30 (1779072270) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00386232 s, 17.0 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (1s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 22:44:31 (1779072271) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (1s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 22:44:32 (1779072272) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00461793 s, 14.2 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (1s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 22:44:33 (1779072273) llite.lustre-ffff923242ae4000.xattr_cache=1 llite.lustre-ffff92325195d800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (3s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 22:44:36 (1779072276) llite.lustre-ffff923242ae4000.xattr_cache=1 llite.lustre-ffff92325195d800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (2s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 22:44:38 (1779072278) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00328717 s, 312 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00353462 s, 290 kB/s 272744: get lock2 272747: get lock1 272747: done 272747: get lock2 272744: done 272744: get lock1 fcntl cmd 7 failed: Resource deadlock avoided 272744: cannot get lock1: Resource deadlock avoided 272747: done 272747: exit rc=0 272744: done 272744: exit rc=0 PASS 74 (6s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 22:44:44 (1779072284) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0196858 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114954 s, 91.2 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4193: 273327 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (7s) == sanityn test 76: Verify MDT open_files listing ======== 22:44:51 (1779072291) opening files: ........ closing files: ........ PASS 76 (18s) == sanityn test 77a: check FIFO NRS policy =============== 22:45:09 (1779072309) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.12864 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906532 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971045 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991355 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884699 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905629 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133475 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147027 s, 7.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141577 s, 7.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141538 s, 7.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff923242ae4000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff923242ae4000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174178 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185599 s, 5.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165047 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174076 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177569 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19768 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170949 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160321 s, 65.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192638 s, 54.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228353 s, 45.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281381 s, 37.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298096 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352448 s, 29.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff923242ae4000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff923242ae4000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317016 s, 33.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0081135 s, 129 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232576 s, 45.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305179 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218371 s, 48.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00762118 s, 138 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156813 s, 66.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402222 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191667 s, 54.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296282 s, 35.4 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg230-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg230-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (3s) == sanityn test 77b: check CRR-N NRS policy ============== 22:45:12 (1779072312) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.127848 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027494 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592253 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852389 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937101 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111804 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104256 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953411 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104375 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116255 s, 9.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118047 s, 8.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120591 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128115 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11912 s, 8.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103406 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113833 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114209 s, 9.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206345 s, 50.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02327 s, 45.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021127 s, 49.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020854 s, 50.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328557 s, 31.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254715 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202728 s, 51.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00862038 s, 122 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279375 s, 37.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254926 s, 41.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409052 s, 25.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319038 s, 32.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123925 s, 84.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264144 s, 39.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00718106 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00610096 s, 172 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff923242ae4000.lock_unused_count=2 removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg230-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg230-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.112826 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311889 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562138 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745401 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712284 s, 14.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff923242ae4000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740959 s, 14.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819557 s, 12.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869513 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715334 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116962 s, 9.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361989 s, 29.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15983 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339429 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16208 s, 6.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276988 s, 37.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201844 s, 51.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222598 s, 47.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163324 s, 6.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff923242ae4000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff923242ae4000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025751 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020331 s, 51.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170076 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414034 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177209 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00781176 s, 134 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00720648 s, 146 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273841 s, 38.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373424 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188784 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183761 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330793 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00963328 s, 109 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185167 s, 56.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292405 s, 35.9 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg230-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg230-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (4s) == sanityn test 77c: check ORR NRS policy ================ 22:45:16 (1779072316) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.141907 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302678 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540334 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789527 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748868 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952083 s, 11.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955206 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764522 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752388 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978343 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0928706 s, 11.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff923242ae4000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805831 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088217 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104174 s, 10.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976426 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966502 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111478 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147909 s, 70.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167795 s, 62.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011955 s, 87.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141634 s, 74.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186513 s, 56.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016213 s, 64.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170858 s, 61.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018019 s, 58.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186698 s, 56.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186862 s, 56.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225713 s, 46.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319446 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00344487 s, 304 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292803 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301245 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246938 s, 42.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff923242ae4000.lock_unused_count=1 removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg230-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg230-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.149104 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261238 s, 40.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443354 s, 23.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547815 s, 19.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744008 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804361 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742401 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835499 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707116 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097144 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105742 s, 9.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff923242ae4000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821019 s, 12.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101706 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105708 s, 9.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885899 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999782 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119107 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142078 s, 73.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197514 s, 53.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021283 s, 49.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165582 s, 63.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00800762 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015332 s, 68.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246503 s, 42.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245292 s, 42.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200722 s, 52.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240206 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250962 s, 41.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181625 s, 57.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00321356 s, 326 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272199 s, 38.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02788 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00566987 s, 185 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff923242ae4000.lock_unused_count=2 removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg230-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg230-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.107829 s, 156 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261521 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718175 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728711 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717327 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653767 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944559 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098007 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811582 s, 12.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923868 s, 11.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986952 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107057 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919565 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101759 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09037 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101012 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101731 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157896 s, 66.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161668 s, 64.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140387 s, 74.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232612 s, 45.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161342 s, 65.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123435 s, 84.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236081 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021364 s, 49.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00535143 s, 196 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011222 s, 93.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0032674 s, 321 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00878833 s, 119 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122186 s, 85.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff923242ae4000.lock_unused_count=2 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113739 s, 92.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163326 s, 64.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285026 s, 36.8 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg230-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg230-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (5s) == sanityn test 77d: check TRR nrs policy ================ 22:45:21 (1779072321) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.146945 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407581 s, 25.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619616 s, 16.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644734 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759764 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769536 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546851 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870523 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584036 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650341 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103534 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137774 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138867 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109207 s, 96.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022671 s, 46.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022498 s, 46.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278037 s, 37.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134734 s, 77.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183842 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191641 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199704 s, 52.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284173 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157887 s, 66.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177829 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145286 s, 72.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142298 s, 73.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440959 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227021 s, 46.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027755 s, 37.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105968 s, 99.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432833 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201136 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270566 s, 38.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff923242ae4000.lock_unused_count=5 removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg230-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg230-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.140614 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034672 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328242 s, 31.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399888 s, 26.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528773 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406633 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554014 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681399 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437324 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669009 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127194 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12128 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124033 s, 8.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143114 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130799 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146868 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151921 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489432 s, 21.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237776 s, 44.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755338 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815996 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465232 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315103 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610688 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797603 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636948 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295972 s, 35.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062381 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364036 s, 28.8 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681228 s, 15.4 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0001-osc-ffff923242ae4000.lock_unused_count=8 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647098 s, 16.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751537 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891563 s, 11.8 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg230-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg230-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.112931 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222866 s, 47.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291954 s, 35.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665542 s, 15.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693266 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894959 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748832 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967311 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958919 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10155 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105091 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11419 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111566 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104537 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107948 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993635 s, 10.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff923242ae4000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff923242ae4000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983307 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122755 s, 85.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00375773 s, 279 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132955 s, 78.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139244 s, 75.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169 s, 62.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029193 s, 35.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00378215 s, 277 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258037 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209032 s, 50.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221281 s, 47.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205655 s, 51.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00699765 s, 150 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00712863 s, 147 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00460813 s, 228 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00471785 s, 222 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff923242ae4000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00547747 s, 191 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg230-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg230-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (5s) == sanityn test 77e: check TBF NID nrs policy ============ 22:45:26 (1779072326) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.202.30@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.173897 s, 96.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415074 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742374 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641284 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076064 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389148 s, 26.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600087 s, 17.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886897 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891449 s, 11.8 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0001-osc-ffff923242ae4000.lock_unused_count=1 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995308 s, 10.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114081 s, 9.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124225 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150563 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14138 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157172 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150367 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17745 s, 5.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157144 s, 66.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194112 s, 54.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153615 s, 68.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00478174 s, 219 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00407553 s, 257 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0048978 s, 214 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00503058 s, 208 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff923242ae4000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00396088 s, 265 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365453 s, 28.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341755 s, 30.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00488748 s, 215 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00506994 s, 207 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343472 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308581 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541356 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00942365 s, 111 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg230-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg230-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.114666 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021779 s, 48.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440018 s, 23.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368545 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561121 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608507 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330042 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521202 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471097 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432192 s, 24.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff923242ae4000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954823 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931889 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839654 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747765 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103527 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083876 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100621 s, 10.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171534 s, 6.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166564 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178387 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172133 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177214 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174468 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186037 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183119 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188907 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177151 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177506 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185108 s, 5.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17893 s, 5.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177195 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190962 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184408 s, 5.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg230-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg230-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.149819 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299115 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548385 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689376 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897394 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101848 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109144 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107122 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140151 s, 7.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff923242ae4000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177164 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179413 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177764 s, 5.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180891 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222499 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20531 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20309 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00507478 s, 207 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236276 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00814889 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00771073 s, 136 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351409 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00569392 s, 184 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00581023 s, 180 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388192 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354423 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431347 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343894 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335355 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331045 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370541 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288402 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355486 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241478 s, 4.3 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg230-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg230-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.13 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435348 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053677 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495774 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06863 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828397 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747576 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752353 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757962 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080067 s, 13.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105348 s, 10.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947529 s, 11.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850284 s, 12.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940069 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100729 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860175 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10049 s, 10.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff923242ae4000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff923242ae4000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160213 s, 65.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147139 s, 71.3 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220269 s, 47.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200941 s, 52.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331205 s, 31.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014795 s, 70.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330623 s, 31.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314965 s, 33.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282931 s, 37.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169418 s, 61.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00528172 s, 199 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101537 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340992 s, 30.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108041 s, 97.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00829341 s, 126 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff923242ae4000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161821 s, 64.8 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg230-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg230-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (7s) == sanityn test 77f: check TBF JobID nrs policy ========== 22:45:33 (1779072333) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg230-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.149968 s, 112 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192383 s, 54.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478283 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403785 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054399 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645855 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604437 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669042 s, 15.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442052 s, 23.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612998 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764121 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869932 s, 12.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10875 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785978 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901755 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105454 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108477 s, 9.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161672 s, 64.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147129 s, 71.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151263 s, 69.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00626663 s, 167 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0092607 s, 113 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190483 s, 55.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166127 s, 63.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240872 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00364991 s, 287 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00389783 s, 269 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255208 s, 41.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229913 s, 45.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232955 s, 45.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00348611 s, 301 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275296 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00517279 s, 203 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg230-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg230-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg230-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.16678 s, 101 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324494 s, 32.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312222 s, 33.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904484 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840564 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872172 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882249 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774582 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105736 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909118 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09511 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111093 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131894 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124398 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111436 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126271 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123428 s, 8.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff923242ae4000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff923242ae4000.lock_unused_count=3 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153587 s, 68.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194629 s, 53.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134892 s, 77.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015531 s, 67.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020137 s, 52.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276773 s, 37.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162457 s, 64.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244207 s, 42.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297941 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325525 s, 32.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0045316 s, 231 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299565 s, 35.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00914019 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012336 s, 85.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00495161 s, 212 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167425 s, 62.6 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg230-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg230-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg230-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.13082 s, 128 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240253 s, 43.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563211 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693448 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831376 s, 12.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950921 s, 11.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085762 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856517 s, 12.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106192 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109265 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00375729 s, 279 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236177 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239208 s, 43.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280472 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277974 s, 37.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407529 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00364579 s, 288 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266684 s, 39.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373646 s, 28.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448451 s, 23.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375816 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493427 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122502 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13778 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff923242ae4000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff923242ae4000.lock_unused_count=1 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123899 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485452 s, 21.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13891 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431314 s, 24.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512215 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00756642 s, 139 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13241 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158922 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107883 s, 9.7 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg230-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg230-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg230-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.175375 s, 95.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350916 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330306 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045026 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593651 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657252 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658217 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08083 s, 13.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905962 s, 11.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951518 s, 11.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff923242ae4000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119255 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122457 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886267 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104988 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927237 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111949 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918634 s, 11.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158679 s, 66.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159404 s, 65.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191121 s, 54.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174129 s, 60.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235662 s, 44.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221611 s, 47.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166243 s, 63.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219279 s, 47.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259526 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230001 s, 45.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225838 s, 46.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207004 s, 50.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230893 s, 45.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268138 s, 39.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214375 s, 48.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00577676 s, 182 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg230-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg230-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (8s) == sanityn test 77g: Change TBF type directly ============ 22:45:41 (1779072341) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (3s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 22:45:44 (1779072344) oleg230-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg230-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg230-client: oleg230-server: ssh exited with exit code 2 oleg230-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg230-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg230-client: oleg230-server: ssh exited with exit code 22 oleg230-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg230-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg230-client: oleg230-server: ssh exited with exit code 22 oleg230-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg230-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg230-client: oleg230-server: ssh exited with exit code 22 PASS 77h (3s) == sanityn test 77i: Change rank of TBF rule ============= 22:45:47 (1779072347) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (8s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 22:45:55 (1779072355) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.127853 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031088 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293649 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716175 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929563 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952848 s, 11.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907811 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105471 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110224 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110323 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958725 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108498 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113628 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095177 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103271 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10101 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947132 s, 11.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251291 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181505 s, 57.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184739 s, 56.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170934 s, 61.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256045 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019583 s, 53.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021551 s, 48.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239192 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248448 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248294 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246489 s, 42.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254484 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263182 s, 39.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261834 s, 40.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267534 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274948 s, 38.1 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg230-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg230-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97534 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4357 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (46s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 22:46:41 (1779072401) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg230-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg230-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg230-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg230-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg230-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.107609 s, 156 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371487 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529282 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642856 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072423 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807584 s, 13.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617579 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770407 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590174 s, 17.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747353 s, 14.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858635 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826292 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795823 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731721 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753585 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604182 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798143 s, 13.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195994 s, 53.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191416 s, 54.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015618 s, 67.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123928 s, 84.6 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157124 s, 66.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247245 s, 42.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255753 s, 41.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157833 s, 66.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230503 s, 45.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171148 s, 61.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211042 s, 49.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029615 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305549 s, 34.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200897 s, 52.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00689818 s, 152 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00203515 s, 515 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg230-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg230-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8341 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9465 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg230-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg230-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg230-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg230-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg230-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.22468 s, 13.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998656 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0956301 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109327 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119642 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101588 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104058 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116544 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.096841 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160869 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161045 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178494 s, 5.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164159 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169129 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162277 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171135 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160037 s, 6.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224179 s, 46.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236116 s, 44.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256017 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254085 s, 41.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227163 s, 46.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306438 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213872 s, 49.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0067754 s, 155 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315103 s, 33.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310682 s, 33.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328634 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302699 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110347 s, 95.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243987 s, 43.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0081524 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00804469 s, 130 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg230-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg230-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8355 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.953 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (135s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 22:48:56 (1779072536) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg230-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg230-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg230-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg230-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg230-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.22829 s, 13.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073737 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074183 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631447 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655274 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714297 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699098 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912075 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870284 s, 12.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154904 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15497 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155068 s, 6.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17325 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154086 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137453 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162189 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165392 s, 6.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142512 s, 73.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213576 s, 49.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208794 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189245 s, 55.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224362 s, 46.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181947 s, 57.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184343 s, 56.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168317 s, 62.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00552674 s, 190 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171194 s, 61.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241404 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218668 s, 48.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204119 s, 51.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222666 s, 47.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208902 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227691 s, 46.1 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg230-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg230-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8346 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9592 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg230-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg230-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg230-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg230-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg230-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.462965 s, 36.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673316 s, 15.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705436 s, 14.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753619 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700267 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743552 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758183 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705829 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104219 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0974347 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120384 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10861 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125448 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109612 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111782 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159904 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140555 s, 7.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278655 s, 37.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236384 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301111 s, 34.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308286 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314533 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258468 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031913 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263463 s, 39.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238371 s, 44.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300245 s, 34.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036356 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265597 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311367 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316239 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301234 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355229 s, 29.5 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg230-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg230-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8303 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.958 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (133s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 22:51:09 (1779072669) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg230-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.25036 s, 67.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243226 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262812 s, 39.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677607 s, 15.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681473 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652148 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837884 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588955 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811126 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833422 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675114 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988445 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821324 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978537 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839692 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877134 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870603 s, 12.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113346 s, 92.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015964 s, 65.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010616 s, 98.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167393 s, 62.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163814 s, 64.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013388 s, 78.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164997 s, 63.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175908 s, 59.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162082 s, 64.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00979185 s, 107 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00903405 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00567243 s, 185 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00353163 s, 297 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0038959 s, 269 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00889639 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268524 s, 39.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg230-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg230-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9601 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73277 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.202.30@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.202.30@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.122885 s, 137 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292828 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065184 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076443 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573619 s, 18.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872938 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526245 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840082 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712743 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825252 s, 12.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088212 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780138 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796999 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082159 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075574 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771176 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921047 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120081 s, 87.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120267 s, 87.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173902 s, 60.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185248 s, 56.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135609 s, 77.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023241 s, 45.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181145 s, 57.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206521 s, 50.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01654 s, 63.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190363 s, 55.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00823151 s, 127 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00447321 s, 234 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00703499 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00323079 s, 325 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00698869 s, 150 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00455177 s, 230 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg230-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg230-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9675 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7311 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.202.30@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg230-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.113306 s, 148 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230129 s, 45.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441421 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251481 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498521 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655285 s, 16.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421706 s, 24.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606929 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677427 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581916 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847545 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847964 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914511 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977904 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.096382 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102571 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10098 s, 10.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0093484 s, 112 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00659542 s, 159 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140507 s, 74.6 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105657 s, 99.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010833 s, 96.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00951827 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124618 s, 84.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178651 s, 58.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013455 s, 77.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00630766 s, 166 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00831673 s, 126 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00965114 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0034946 s, 300 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00399296 s, 263 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00358555 s, 292 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00931523 s, 113 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg230-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg230-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9734 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.88176 s, 21.5 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg230-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.459209 s, 36.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312955 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448148 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557623 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492901 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405523 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689639 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549877 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887362 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079803 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101259 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781781 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973574 s, 10.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291879 s, 3.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311099 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355778 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363502 s, 2.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019404 s, 54.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191059 s, 54.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182331 s, 57.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024308 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236943 s, 44.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293126 s, 35.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021447 s, 48.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317708 s, 33.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132964 s, 78.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105145 s, 99.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335935 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00477501 s, 220 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346168 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00369849 s, 284 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129987 s, 80.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00638001 s, 164 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg230-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg230-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.85455 s, 10.6 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74081 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.91142 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73287 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg230-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.135723 s, 124 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445213 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726428 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966264 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138075 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141106 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169337 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169454 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142462 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147962 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147931 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159303 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180148 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18859 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168972 s, 6.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166981 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183638 s, 5.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106759 s, 98.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183915 s, 57.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019444 s, 53.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255118 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197734 s, 53.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215692 s, 48.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229138 s, 45.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221608 s, 47.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131469 s, 79.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199876 s, 52.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00913676 s, 115 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279162 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254745 s, 41.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00795847 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00368522 s, 285 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00670186 s, 156 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg230-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg230-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6701 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.435 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg230-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.131059 s, 128 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422626 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390233 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151494 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151284 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154567 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166932 s, 6.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150686 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16048 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176939 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187652 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160844 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161695 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167612 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161839 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15749 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182922 s, 5.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016891 s, 62.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185751 s, 56.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224142 s, 46.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211952 s, 49.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168726 s, 62.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166005 s, 63.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183908 s, 57.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219504 s, 47.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168054 s, 62.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177723 s, 59.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214121 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00468554 s, 224 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00484008 s, 217 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00507404 s, 207 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00553089 s, 190 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0037776 s, 278 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg230-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg230-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.98902 s, 21.0 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73239 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg230-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.122051 s, 137 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025232 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691954 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777572 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805967 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923577 s, 11.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875262 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805844 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892125 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786684 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882509 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930506 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867028 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939129 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836291 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101472 s, 10.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162947 s, 64.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022596 s, 46.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249952 s, 42.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164449 s, 63.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220401 s, 47.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164404 s, 63.8 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022728 s, 46.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217724 s, 48.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203898 s, 51.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231331 s, 45.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268149 s, 39.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227379 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228994 s, 45.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272151 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024468 s, 42.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264947 s, 39.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg230-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg230-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77202 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73125 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7665 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73143 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (357s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 22:57:06 (1779073026) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg230-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.254652 s, 65.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293993 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552057 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530486 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066382 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698474 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998524 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106529 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867681 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977117 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993149 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107354 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120143 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10052 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124248 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120934 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115655 s, 9.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164626 s, 63.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164736 s, 63.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194263 s, 54.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016744 s, 62.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185247 s, 56.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01879 s, 55.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200819 s, 52.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184926 s, 56.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166115 s, 63.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023722 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234234 s, 44.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014567 s, 72.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00538881 s, 195 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00635885 s, 165 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.003021 s, 347 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00453514 s, 231 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg230-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg230-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94056 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73695 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (36s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 22:57:42 (1779073062) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.202.30@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.202.30@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.155944 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0682307 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779624 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614617 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785893 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703242 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634762 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934878 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903852 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137073 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120386 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150488 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141513 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1494 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144736 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144871 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163951 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01157 s, 90.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187605 s, 55.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144097 s, 72.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177132 s, 59.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169469 s, 61.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178486 s, 58.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00593738 s, 177 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194269 s, 54.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00385369 s, 272 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200085 s, 52.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00580728 s, 181 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139279 s, 75.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263364 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00327058 s, 321 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295661 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00448301 s, 234 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg230-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg230-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.92868 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.733 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (37s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 22:58:19 (1779073099) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.202.30@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg230-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.111361 s, 151 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195552 s, 53.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048922 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513478 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408607 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700827 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732536 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688035 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590552 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695997 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927755 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104104 s, 10.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120025 s, 8.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110176 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122975 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112258 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130032 s, 8.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110615 s, 94.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00901266 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01993 s, 52.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217597 s, 48.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203803 s, 51.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236742 s, 44.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211908 s, 49.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235317 s, 44.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133343 s, 78.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209299 s, 50.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206718 s, 50.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00379391 s, 276 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00428226 s, 245 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257421 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110101 s, 95.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00261364 s, 401 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg230-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg230-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97313 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97229 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (34s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 22:58:53 (1779073133) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg230-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.477585 s, 35.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360605 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444902 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558709 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519369 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503651 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540438 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343213 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819273 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082489 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994994 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937693 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887613 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290047 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298178 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363819 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365417 s, 2.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122378 s, 85.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198363 s, 52.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187857 s, 55.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146548 s, 71.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156949 s, 66.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176725 s, 59.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188766 s, 55.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104954 s, 99.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233607 s, 44.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191193 s, 54.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00753909 s, 139 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216177 s, 48.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025108 s, 41.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00485071 s, 216 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00706172 s, 148 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00201052 s, 522 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg230-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg230-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.79054 s, 10.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73188 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.91204 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73228 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (77s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 23:00:10 (1779073210) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg230-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.848101 s, 19.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033678 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618305 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675426 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717206 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985936 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902426 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098618 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866053 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161312 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156181 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161835 s, 6.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178773 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175296 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167065 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176249 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176125 s, 6.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136529 s, 76.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205822 s, 50.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158184 s, 66.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00938811 s, 112 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157774 s, 66.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178648 s, 58.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244199 s, 42.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00753108 s, 139 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016685 s, 62.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207469 s, 50.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00425304 s, 247 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240564 s, 43.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00598537 s, 175 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00743992 s, 141 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255206 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00458507 s, 229 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg230-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg230-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6901 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9047 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (63s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 23:01:13 (1779073273) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg230-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.125658 s, 134 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346914 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482016 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507736 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775979 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122239 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117836 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12781 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149086 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151375 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185334 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165142 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172621 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173038 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146295 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182872 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173775 s, 6.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248563 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224585 s, 46.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226875 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234364 s, 44.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317624 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352248 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362053 s, 29.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186989 s, 56.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404444 s, 25.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193565 s, 54.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236971 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377183 s, 27.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156948 s, 66.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205832 s, 50.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018498 s, 56.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036487 s, 28.7 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg230-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg230-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9493 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.75041 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg230-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.214275 s, 78.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442484 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366894 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135416 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132765 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135206 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174772 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181013 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165442 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16938 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158405 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168785 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180859 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17504 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170744 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186306 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179689 s, 5.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024884 s, 42.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276334 s, 37.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271135 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304204 s, 34.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243961 s, 43.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238716 s, 43.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279008 s, 37.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316814 s, 33.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271418 s, 38.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031987 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00836182 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00393418 s, 267 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00976721 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173105 s, 60.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00822967 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00163453 s, 642 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg230-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg230-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.81959 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73653 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77087 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7379 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (112s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 23:03:05 (1779073385) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (3s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 23:03:08 (1779073388) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg230-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.65622 s, 287 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg230-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0221 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (51s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 23:03:59 (1779073439) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.123452 s, 136 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377986 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107817 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11756 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144341 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152753 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149235 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171771 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180329 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162208 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159452 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170315 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178093 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174885 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167601 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162939 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167741 s, 6.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff923242ae4000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff923242ae4000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00745274 s, 141 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156186 s, 67.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245322 s, 42.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250244 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260151 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349859 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135408 s, 77.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385339 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112434 s, 93.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315699 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00625219 s, 168 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225915 s, 46.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00699045 s, 150 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106852 s, 98.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00298174 s, 352 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00252344 s, 416 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg230-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg230-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97887 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97009 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.109776 s, 153 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308488 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432815 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126336 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153803 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15588 s, 6.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143744 s, 7.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142398 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167276 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141454 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158477 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16968 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146398 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140401 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145888 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154303 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167246 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184632 s, 56.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217728 s, 48.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196322 s, 53.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181211 s, 57.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160801 s, 65.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174337 s, 60.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246477 s, 42.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0088101 s, 119 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00702441 s, 149 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163188 s, 64.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00722345 s, 145 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034264 s, 30.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280381 s, 37.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244481 s, 42.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00656428 s, 160 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138568 s, 75.7 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg230-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg230-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.01994 s, 20.9 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97878 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (65s) == sanityn test 77o: Changing rank should not panic ====== 23:05:04 (1779073504) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (4s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 23:05:09 (1779073509) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (40s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 23:05:48 (1779073548) mds.MDS.mdt.nrs_policies=tbf oleg230-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg230-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg230-client: oleg230-server: ssh exited with exit code 36 oleg230-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg230-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg230-client: oleg230-server: ssh exited with exit code 22 oleg230-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg230-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg230-client: oleg230-server: ssh exited with exit code 22 oleg230-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg230-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg230-client: oleg230-server: ssh exited with exit code 22 oleg230-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg230-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg230-client: oleg230-server: ssh exited with exit code 22 oleg230-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg230-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg230-client: oleg230-server: ssh exited with exit code 22 oleg230-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg230-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg230-client: oleg230-server: ssh exited with exit code 22 oleg230-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg230-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg230-client: oleg230-server: ssh exited with exit code 22 oleg230-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg230-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg230-client: oleg230-server: ssh exited with exit code 22 oleg230-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg230-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg230-client: oleg230-server: ssh exited with exit code 22 oleg230-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg230-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg230-client: oleg230-server: ssh exited with exit code 22 oleg230-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg230-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg230-client: oleg230-server: ssh exited with exit code 22 oleg230-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg230-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg230-client: oleg230-server: ssh exited with exit code 22 oleg230-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg230-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg230-client: oleg230-server: ssh exited with exit code 22 oleg230-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg230-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg230-client: oleg230-server: ssh exited with exit code 22 oleg230-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg230-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg230-client: oleg230-server: ssh exited with exit code 22 oleg230-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg230-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg230-client: oleg230-server: ssh exited with exit code 22 oleg230-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg230-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg230-client: oleg230-server: ssh exited with exit code 22 oleg230-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg230-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg230-client: oleg230-server: ssh exited with exit code 22 oleg230-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg230-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg230-client: oleg230-server: ssh exited with exit code 22 oleg230-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg230-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg230-client: oleg230-server: ssh exited with exit code 22 oleg230-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg230-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg230-client: oleg230-server: ssh exited with exit code 22 oleg230-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg230-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg230-client: oleg230-server: ssh exited with exit code 22 oleg230-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg230-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg230-client: oleg230-server: ssh exited with exit code 22 oleg230-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg230-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg230-client: oleg230-server: ssh exited with exit code 22 oleg230-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg230-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg230-client: oleg230-server: ssh exited with exit code 22 oleg230-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg230-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg230-client: oleg230-server: ssh exited with exit code 22 oleg230-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg230-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg230-client: oleg230-server: ssh exited with exit code 22 oleg230-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg230-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg230-client: oleg230-server: ssh exited with exit code 22 oleg230-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg230-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg230-client: oleg230-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (12s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 23:06:00 (1779073560) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 309081 kill md_thread xargs PASS 77r (41s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 23:06:41 (1779073601) oleg230-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg230-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg230-client: oleg230-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (3s) == sanityn test 79: xattr: intent error ================== 23:06:44 (1779073604) File: /mnt/lustre/d79.sanityn Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272506413 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-17 23:06:45.000000000 -0400 Modify: 2026-05-17 23:06:45.000000000 -0400 Change: 2026-05-17 23:06:45.000000000 -0400 Birth: 2026-05-17 23:06:44.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (13s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 23:06:57 (1779073617) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (1s) == sanityn test 80b: Accessing directory during migration ========================================================== 23:06:58 (1779073618) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (1s) == sanityn test 81a: rename and stat under striped directory ========================================================== 23:06:59 (1779073619) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (1s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 23:07:00 (1779073620) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (1s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 23:07:01 (1779073621) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (1s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 23:07:02 (1779073622) total: 2500 open/close in 6.32 seconds: 395.45 ops/second snapshot_time 1779073649.007155724 secs.nsecs start_time 1779073622.723042333 secs.nsecs elapsed_time 26.284113391 secs.nsecs open 2505 samples [usecs] 90 3031 840893 321731401 close 2504 samples [usecs] 66 676 270582 33747192 mknod 2501 samples [usecs] 174 2908 728838 245789360 unlink 2500 samples [usecs] 280 11965 1200595 1234517471 mkdir 2 samples [usecs] 955 1728 2683 3898009 rmdir 2 samples [usecs] 1850 2943 4793 12083749 rename 2500 samples [usecs] 251 8714 3442886 7287041136 getattr 3760 samples [usecs] 8 834 83085 3664775 parallel_rename_file 2500 samples [usecs] 251 8714 3442886 7287041136 crossdir_rename 2500 samples [usecs] 251 8714 3442886 7287041136 PASS 81d (29s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 23:07:31 (1779073651) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7571 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7571 PASS 82 (2s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 23:07:33 (1779073653) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (1s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 23:07:34 (1779073654) fail_loc=0x8000060b PASS 84 (8s) == sanityn test 85: Lustre API root cache race =========== 23:07:42 (1779073662) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1779073662 Finishing test test1 at 1779073663 PASS 85 (3s) == sanityn test 90: open/create and unlink striped directory ========================================================== 23:07:45 (1779073665) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (1s) == sanityn test 91: chmod and unlink striped directory === 23:07:46 (1779073666) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (1s) == sanityn test 92: create remote directory under orphan directory ========================================================== 23:07:47 (1779073667) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (1s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 23:07:48 (1779073668) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5742 0x166e 0x280000400 0 5766 0x1686 0x240000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5767 0x1687 0x240000400 1 5743 0x166f 0x280000400 rc1=0 and rc2=0 PASS 93 (8s) == sanityn test 94: signal vs CP callback race =========== 23:07:56 (1779073676) osc.lustre-OST0000-osc-ffff923242ae4000.idle_timeout=0 osc.lustre-OST0000-osc-ffff92325195d800.idle_timeout=0 osc.lustre-OST0001-osc-ffff923242ae4000.idle_timeout=0 osc.lustre-OST0001-osc-ffff92325195d800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0422501 s, 194 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.09441 s, 1.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5988: 326841 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0143779 s, 285 kB/s osc.lustre-OST0000-osc-ffff923242ae4000.idle_timeout=debug osc.lustre-OST0000-osc-ffff92325195d800.idle_timeout=debug osc.lustre-OST0001-osc-ffff923242ae4000.idle_timeout=debug osc.lustre-OST0001-osc-ffff92325195d800.idle_timeout=debug PASS 94 (11s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 23:08:07 (1779073687) llite.lustre-ffff923242ae4000.fast_read=0 llite.lustre-ffff92325195d800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0168727 s, 124 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff923242ae4000.fast_read=1 llite.lustre-ffff92325195d800.fast_read=1 PASS 95a (14s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 23:08:21 (1779073701) llite.lustre-ffff923242ae4000.fast_read=0 llite.lustre-ffff92325195d800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00189492 s, 6.5 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff923242ae4000.fast_read=1 llite.lustre-ffff92325195d800.fast_read=1 PASS 95b (8s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 23:08:29 (1779073709) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (1s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 23:08:30 (1779073710) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00138195 s, 3.0 MB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (2s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 23:08:32 (1779073712) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0747998 s, 28.0 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (2s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 23:08:34 (1779073714) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0510839 s, 41.1 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (2s) == sanityn test 100e: DoM: read on open and file size ==== 23:08:36 (1779073716) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00228536 s, 5.3 MB/s PASS 100e (2s) == sanityn test 101a: Discard DoM data on unlink ========= 23:08:38 (1779073718) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000584397 s, 7.0 MB/s PASS 101a (2s) == sanityn test 101b: Discard DoM data on rename ========= 23:08:40 (1779073720) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000746152 s, 5.5 MB/s PASS 101b (3s) == sanityn test 101c: Discard DoM data on close-unlink === 23:08:43 (1779073723) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00040402 s, 10.1 MB/s PASS 101c (3s) SKIP: sanityn test_102 skipping ALWAYS excluded test 102 == sanityn test 103: Test size correctness with lockahead ========================================================== 23:08:46 (1779073726) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1779073727 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1779073730 Finishing test test23 at 1779073730 PASS 103 (7s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 23:08:53 (1779073733) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (2s) == sanityn test 105: Glimpse and lock cancel race ======== 23:08:55 (1779073735) fail_loc=0x416 PASS 105 (18s) == sanityn test 106a: Verify the btime via statx() ======= 23:09:13 (1779073753) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (1s) == sanityn test 106b: Glimpse RPCs test for statx ======== 23:09:14 (1779073754) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00324719 s, 315 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205272508948 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-17 23:09:14.000000000 -0400 Modify: 2026-05-17 23:09:14.000000000 -0400 Change: 2026-05-17 23:09:14.000000000 -0400 Birth: 2026-05-17 23:09:14.000000000 -0400 /mnt/lustre/f106b.sanityn 144115205272508948 -rw-r--r-- 1 0 0 1779073754 1779073754 1779073754 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205272508948 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-17 23:09:14.000000000 -0400 Modify: 2026-05-17 23:09:14.000000000 -0400 Change: 2026-05-17 23:09:14.000000000 -0400 Birth: 2026-05-17 23:09:14.000000000 -0400 1779073754 1024 1 PASS 106b (2s) == sanityn test 106c: Verify statx attributes mask ======= 23:09:16 (1779073756) PASS 106c (3s) == sanityn test 107a: Basic grouplock conflict =========== 23:09:19 (1779073759) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0872659 s, 120 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5749 0x1675 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7571 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7571 First grouplock blocks second one PASS 107a (4s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 23:09:23 (1779073763) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0924138 s, 113 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5776 0x1690 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7571 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (8s) == sanityn test 108a: lseek: parallel updates ============ 23:09:31 (1779073771) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00158437 s, 6.3 MB/s llite.lustre-ffff923242ae4000.hybrid_io=0 llite.lustre-ffff92325195d800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.37118 s, 1.9 MB/s PASS 108a (7s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 23:09:38 (1779073778) 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) Starting race between client mount instances (50 iterations): Iteration 1 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg230-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 2 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 3 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 4 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 5 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 6 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 7 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 8 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg230-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 9 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 10 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg230-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 11 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 12 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 13 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 14 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 15 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 16 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg230-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 17 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg230-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 18 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 19 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 20 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg230-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 21 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 22 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg230-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 23 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg230-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 24 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg230-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 25 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 26 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 27 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 28 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 29 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg230-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 30 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg230-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 31 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg230-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 32 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg230-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 33 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 34 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 35 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 36 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 37 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 38 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg230-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 39 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 40 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 41 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg230-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 42 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 43 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 44 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 45 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg230-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 46 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 47 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg230-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 48 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 49 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg230-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Iteration 50 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 192.168.202.130@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg230-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 PASS 109 (553s) == sanityn test 110: do not grant another lock on resend ========================================================== 23:18:51 (1779074331) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205306056707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-17 23:18:51.000000000 -0400 Modify: 2026-05-17 23:18:51.000000000 -0400 Change: 2026-05-17 23:18:51.000000000 -0400 Birth: 2026-05-17 23:18:51.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg230-client.virtnet oleg230-client.virtnet PASS 110 (65s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 23:19:56 (1779074396) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (2s) == sanityn test 112: update max-inherit in default LMV === 23:19:58 (1779074398) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (1s) == sanityn test 113: check servers of specified fs ======= 23:19:59 (1779074399) lustre-OST0000-osc-ffff923243278800 active. lustre-OST0001-osc-ffff923243278800 active. lustre-OST0000-osc-ffff923243278800 active. lustre-OST0001-osc-ffff923243278800 active. MGC192.168.202.130@tcp active. PASS 113 (4s) == sanityn test 114: implicit default LMV inherit ======== 23:20:03 (1779074403) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (1s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 23:20:04 (1779074404) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (1s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 23:20:05 (1779074405) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (2s) == sanityn test 117a: TCU: Init and enable Trash Can on MDTs ========================================================== 23:20:07 (1779074407) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_enable=0 PASS 117a (5s) == sanityn test 117b: Move regular file and empty dir into trash can dir ========================================================== 23:20:12 (1779074412) SKIP: sanityn test_117b ldiskfs only test SKIP 117b (1s) == sanityn test 117c: Move deleted tree with multiple levels into trash ========================================================== 23:20:13 (1779074413) SKIP: sanityn test_117c ldiskfs only test SKIP 117c (2s) == sanityn test 117d: Per-User Trash can Type testing ==== 23:20:15 (1779074415) SKIP: sanityn test_117d ldiskfs only test SKIP 117d (2s) == sanityn test 117e: Undeleted dir in trash should keep its original xattrs ========================================================== 23:20:17 (1779074417) SKIP: sanityn test_117e ldiskfs only test SKIP 117e (1s) == sanityn test 117f: Uncache the dentry under the trash dir ========================================================== 23:20:18 (1779074418) SKIP: sanityn test_117f ldiskfs only test SKIP 117f (2s) == sanityn test 117g: Access .Trash for a non-striped directory ========================================================== 23:20:20 (1779074420) SKIP: sanityn test_117g ldiskfs only test SKIP 117g (1s) == sanityn test 117h: Permission testing of various operations for files in Trash Can ========================================================== 23:20:21 (1779074421) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain Trash Can Type: plain Trash list: ===> /mnt/lustre/.lustre/.ltrash/MDT0000: 0x200000403:0x5:0x0 /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000403:0x5:0x0: tdir1 tfile3 /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000403:0x5:0x0/tdir1: <=== touch: setting times of '/mnt/lustre/.lustre/.ltrash/f117h.sanityn.crfail': No such file or directory mkdir: cannot create directory '/mnt/lustre/.lustre/.ltrash/f117h.sanityn.crfail': Operation not permitted touch: setting times of '/mnt/lustre/.lustre/.ltrash/MDT0000/f117h.sanityn.crfail': No such file or directory mkdir: cannot create directory '/mnt/lustre/.lustre/.ltrash/MDT0000/f117h.sanityn.crfail': Operation not permitted rmdir: failed to remove '/mnt/lustre/.lustre/.ltrash/MDT0000/0x200000403:0x5:0x0': Operation not permitted touch: setting times of '/mnt/lustre/.lustre/.ltrash/MDT0000/0x200000403:0x5:0x0/f117h.sanityn.crfail': No such file or directory mv: cannot move '/mnt/lustre/d117h.sanityn/f117h.sanityn.mvfail' to '/mnt/lustre/.lustre/.ltrash/f117h.sanityn.mvfail': Operation not permitted mv: cannot move '/mnt/lustre/d117h.sanityn/f117h.sanityn.mvfail' to '/mnt/lustre/.lustre/.ltrash/MDT0000/f117h.sanityn.mvfail': No data available mv: cannot move '/mnt/lustre/d117h.sanityn/f117h.sanityn.mvfail' to '/mnt/lustre/.lustre/.ltrash/MDT0000/0x200000403:0x5:0x0/f117h.sanityn.mvfail': No data available mv: cannot move '/mnt/lustre/d117h.sanityn/d117h.sanityn.mvfail' to '/mnt/lustre/.lustre/.ltrash/d117h.sanityn.mvfail': Operation not permitted mv: cannot move '/mnt/lustre/d117h.sanityn/d117h.sanityn.mvfail' to '/mnt/lustre/.lustre/.ltrash/MDT0000/d117h.sanityn.mvfail': No data available mv: cannot move '/mnt/lustre/d117h.sanityn/d117h.sanityn.mvfail' to '/mnt/lustre/.lustre/.ltrash/MDT0000/0x200000403:0x5:0x0/d117h.sanityn.mvfail': No data available chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000403:0x5:0x0//tfile3 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000403:0x5:0x0//tfile3 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000403:0x5:0x0//tfile3 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000403:0x5:0x0//tfile3 chmod: changing permissions of '/mnt/lustre/.lustre/.ltrash/MDT0000/0x200000403:0x5:0x0//tfile3': Bad file descriptor truncate: cannot truncate '/mnt/lustre/.lustre/.ltrash/MDT0000/0x200000403:0x5:0x0//tfile3' to length 4096: Bad file descriptor chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000403:0x5:0x0//tdir1 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000403:0x5:0x0//tdir1 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000403:0x5:0x0//tdir1 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000403:0x5:0x0//tdir1 chmod: changing permissions of '/mnt/lustre/.lustre/.ltrash/MDT0000/0x200000403:0x5:0x0//tdir1': Bad file descriptor setfattr: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000403:0x5:0x0//tfile3: Bad file descriptor setfattr: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000403:0x5:0x0//tfile3: Bad file descriptor setfattr: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000403:0x5:0x0//tdir1: Bad file descriptor setfattr: /mnt/lustre/.lustre/.ltrash/MDT0000/0x200000403:0x5:0x0//tdir1: Bad file descriptor open: Bad file descriptor open: Bad file descriptor read: Operation not permitted mmap: Operation not permitted open: Is a directory open: Is a directory mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid PASS 117h (8s) == sanityn test 117i: Forbid to create a file with name .Trash ========================================================== 23:20:29 (1779074429) mkdir: cannot create directory '/mnt/lustre/d117i.sanityn/.Trash': Invalid argument touch: setting times of '/mnt/lustre/d117i.sanityn/.Trash': No such file or directory open(O_RDWR|O_CREAT): Invalid argument mcreate: cannot create `/mnt/lustre/d117i.sanityn/.Trash' with mode 0100644: Invalid argument ln: failed to create hard link '/mnt/lustre/d117i.sanityn/.Trash' => '/mnt/lustre/d117i.sanityn/f117i.sanityn': Invalid argument ln: failed to create symbolic link '/mnt/lustre/d117i.sanityn/.Trash': Invalid argument mv: cannot move '/mnt/lustre/d117i.sanityn/f117i.sanityn' to a subdirectory of itself, '/mnt/lustre/d117i.sanityn/.Trash' mv: cannot move '/mnt/lustre/d117i.sanityn/f117i.sanityn' to a subdirectory of itself, '/mnt/lustre/.Trash' ln: failed to create symbolic link '/mnt/lustre/d117i.sanityn/.Trash': Invalid argument mv: cannot move '/mnt/lustre/d117i.sanityn/d117i.sanityn' to a subdirectory of itself, '/mnt/lustre/d117i.sanityn/.Trash' mv: cannot move '/mnt/lustre/d117i.sanityn/d117i.sanityn' to a subdirectory of itself, '/mnt/lustre/.Trash' PASS 117i (3s) == sanityn test 117j: Move back file from Trash Can ====== 23:20:32 (1779074432) SKIP: sanityn test_117j ldiskfs only test SKIP 117j (1s) == sanityn test 117k: Unrm file from Trash Can =========== 23:20:34 (1779074434) SKIP: sanityn test_117k ldiskfs only test SKIP 117k (1s) == sanityn test 117l: Clean up files in Trash Can via command 'lfs trash clean' ========================================================== 23:20:35 (1779074435) SKIP: sanityn test_117l ldiskfs only test SKIP 117l (2s) == sanityn test 117m: Empty Trash Can via the command 'lfs trash clean' ========================================================== 23:20:37 (1779074437) SKIP: sanityn test_117m ldiskfs only test SKIP 117m (1s) == sanityn test 117n: Update LinkEA accordingly when moving file into Trash Can ========================================================== 23:20:38 (1779074438) SKIP: sanityn test_117n ldiskfs only test SKIP 117n (2s) == sanityn test 117o: Set/Clear NOTRASH flag on file ===== 23:20:40 (1779074440) SKIP: sanityn test_117o ldiskfs only test SKIP 117o (2s) == sanityn test 117p: Trash Can handling for remote dir/file ========================================================== 23:20:42 (1779074442) SKIP: sanityn test_117p needs >= 2 MDTs SKIP 117p (1s) == sanityn test 117pb: Trash Can handling for remote dir/file with multiple levels ========================================================== 23:20:44 (1779074444) SKIP: sanityn test_117pb needs >= 2 MDTs SKIP 117pb (2s) == sanityn test 117q: Access Trash Can via '.Trash' for striped directory ========================================================== 23:20:45 (1779074445) SKIP: sanityn test_117q needs >= 2 MDTs SKIP 117q (2s) == sanityn test 117r: Verify the deleting a striped directory on Trash Can ========================================================== 23:20:47 (1779074447) SKIP: sanityn test_117r ldiskfs only test SKIP 117r (1s) == sanityn test 117s: Create trash shard if it does not exist ========================================================== 23:20:49 (1779074449) SKIP: sanityn test_117s ldiskfs only test SKIP 117s (2s) == sanityn test 117t: UID/GID/ProjID for files in Trash Can ========================================================== 23:20:50 (1779074450) SKIP: sanityn test_117t ldiskfs only test SKIP 117t (2s) == sanityn test 117u: statfs() call should not account the space in Trash Can ========================================================== 23:20:52 (1779074452) SKIP: sanityn test_117u ldiskfs only test SKIP 117u (2s) == sanityn test 117ub: statfs() account the inode usage in Trash Can ========================================================== 23:20:54 (1779074454) SKIP: sanityn test_117ub ldiskfs only test SKIP 117ub (1s) == sanityn test 117v: Test for Trash Can UID/GID/PROJID setting ========================================================== 23:20:55 (1779074455) SKIP: sanityn test_117v ldiskfs only test SKIP 117v (3s) == sanityn test 117w: Test Trash Can with original UID/GID/PROJID ========================================================== 23:20:58 (1779074458) SKIP: sanityn test_117w ldiskfs only test SKIP 117w (1s) == sanityn test 117x: Unrm Files from Trash Can should also restore UID/GID/PROJID ========================================================== 23:20:59 (1779074459) SKIP: sanityn test_117x ldiskfs only test SKIP 117x (2s) == sanityn test 117ya: Test with symbol links with Trash Can ========================================================== 23:21:01 (1779074461) SKIP: sanityn test_117ya ldiskfs only test SKIP 117ya (1s) == sanityn test 117yb: Compare the directories containing symlinks after unrm recover ========================================================== 23:21:02 (1779074462) SKIP: sanityn test_117yb ldiskfs only test SKIP 117yb (2s) == sanityn test 117yc: Restoring UID/GID/PROJID for symlinks works ========================================================== 23:21:04 (1779074464) SKIP: sanityn test_117yc ldiskfs only test SKIP 117yc (1s) == sanityn test 117za: Test Trash Can for UID Trash Can type with nodemap ========================================================== 23:21:05 (1779074465) SKIP: sanityn test_117za ldiskfs only test SKIP 117za (1s) == sanityn test 117zb: Test Trash Can for PLAIN Trash Can type with nodemap ========================================================== 23:21:06 (1779074466) SKIP: sanityn test_117zb ldiskfs only test SKIP 117zb (2s) == sanityn test 117zc: TCU: Test striped dir for UID Trash Can type ========================================================== 23:21:08 (1779074468) SKIP: sanityn test_117zc ldiskfs only test SKIP 117zc (1s) == sanityn test 117zd: TCU: Test striped dir for UID Trash Can type with nodemap ========================================================== 23:21:10 (1779074470) On MGS 192.168.202.130, active = nodemap.active=1 waiting 10 secs for sync On MGS 192.168.202.130, tnm.trusted_nodemap = nodemap.tnm.trusted_nodemap=1 waiting 10 secs for sync nodemap.tnm.idmap= [ { idtype: uid, client_id: 500, fs_id: 10000 }, { idtype: uid, client_id: 501, fs_id: 10001 }, { idtype: uid, client_id: 502, fs_id: 10002 }, { idtype: uid, client_id: 503, fs_id: 10003 }, { idtype: uid, client_id: 504, fs_id: 10004 }, { idtype: uid, client_id: 505, fs_id: 10005 }, { idtype: uid, client_id: 506, fs_id: 10006 }, { idtype: uid, client_id: 507, fs_id: 10007 }, { idtype: uid, client_id: 508, fs_id: 10008 }, { idtype: uid, client_id: 509, fs_id: 10009 } ] SKIP: sanityn test_117zd ldiskfs only test On MGS 192.168.202.130, active = nodemap.active=0 waiting 10 secs for sync SKIP 117zd (40s) == sanityn test 118Aa: TCU: Orphan object handling for a regular file or an empty dir ========================================================== 23:21:49 (1779074509) SKIP: sanityn test_118Aa ldiskfs only test SKIP 118Aa (2s) == sanityn test 118Ab: TCU: Orphan object handling for non-emtpy dir ========================================================== 23:21:51 (1779074511) SKIP: sanityn test_118Ab ldiskfs only test SKIP 118Ab (1s) == sanityn test 118Ba: TCU: 'oldest' repeated deletion for a regular file or empty dir ========================================================== 23:21:52 (1779074512) SKIP: sanityn test_118Ba ldiskfs only test SKIP 118Ba (2s) == sanityn test 118Bb: TCU: 'timestamp' repeated deletion policy ========================================================== 23:21:54 (1779074514) SKIP: sanityn test_118Bb ldiskfs only test SKIP 118Bb (2s) == sanityn test 118C: TCU: add support for rename() ====== 23:21:56 (1779074516) SKIP: sanityn test_118C ldiskfs only test SKIP 118C (2s) == sanityn test 118d: TCU: repeated deletion of same fname for a empty dir in DNE env ========================================================== 23:21:58 (1779074518) SKIP: sanityn test_118d ldiskfs only test SKIP 118d (2s) == sanityn test 118e: TCU: Repeated deletion of same name for a non-empty remote dir ========================================================== 23:22:00 (1779074520) SKIP: sanityn test_118e ldiskfs only test SKIP 118e (2s) == sanityn test 118f: Avoid caching negative dentry cache for dir in Trash Can ========================================================== 23:22:02 (1779074522) SKIP: sanityn test_118f ldiskfs only test SKIP 118f (1s) == sanityn test 118g: TCU: Tolerate ENOSPC error bypass Trash Can ========================================================== 23:22:03 (1779074523) SKIP: sanityn test_118g ldiskfs only test SKIP 118g (2s) == sanityn test 118h: TCU: run mdtest with TCU enabled === 23:22:05 (1779074525) SKIP: sanityn test_118h mdtest not found SKIP 118h (1s) == sanityn test 118i: TCU: trusted.unrm xattr should not be cached on client-side ========================================================== 23:22:06 (1779074526) SKIP: sanityn test_118i ldiskfs only test SKIP 118i (1s) == sanityn test 118j: TCU: access .Trash from users with different UID ========================================================== 23:22:08 (1779074528) SKIP: sanityn test_118j ldiskfs only test SKIP 118j (2s) == sanityn test 118k: TCU: acess .Trash from users with different NODEMAP UIDs ========================================================== 23:22:09 (1779074529) SKIP: sanityn test_118k ldiskfs only test SKIP 118k (2s) == sanityn test 118l: Access global trash dir via LU_TRASH_FID with subdir mount ========================================================== 23:22:11 (1779074531) SKIP: sanityn test_118l ldiskfs only test SKIP 118l (2s) == sanityn test 118m: Access .ltrash under normal root dir and subdir mount point ========================================================== 23:22:13 (1779074533) SKIP: sanityn test_118m ldiskfs only test SKIP 118m (1s) == sanityn test 118n: lfs trash state should return state for a symlink file correctly ========================================================== 23:22:14 (1779074534) SKIP: sanityn test_118n ldiskfs only test SKIP 118n (2s) == sanityn test 118o: lfs trash unrm should work with relative path ========================================================== 23:22:16 (1779074536) SKIP: sanityn test_118o ldiskfs only test SKIP 118o (1s) == sanityn test 118p: Setting notrash flag on symlink file ========================================================== 23:22:17 (1779074537) SKIP: sanityn test_118p ldiskfs only test SKIP 118p (2s) == sanityn test 118q: The mode of a directory moved into Trash should not be changed ========================================================== 23:22:19 (1779074539) SKIP: sanityn test_118q ldiskfs only test SKIP 118q (1s) == sanityn test 118r: TCU: non-root user access Trash Can and restore files ========================================================== 23:22:20 (1779074540) SKIP: sanityn test_118r ldiskfs only test SKIP 118r (1s) == sanityn test 118s: TCU: The FS flags of a file or directory should be kept ========================================================== 23:22:21 (1779074541) SKIP: sanityn test_118s ldiskfs only test SKIP 118s (2s) == sanityn test 118t: TCU: unrmed directory should keep LUSTRE_PROJHERIT_FL flag ========================================================== 23:22:23 (1779074543) SKIP: sanityn test_118t ldiskfs only test SKIP 118t (1s) == sanityn test 118u: TCU: Space usage of deleted files == 23:22:24 (1779074544) SKIP: sanityn test_118u ldiskfs only test SKIP 118u (2s) == sanityn test 118v: TCU: Trash Can for encrypted files ========================================================== 23:22:26 (1779074546) SKIP: sanityn test_118v ldiskfs only test SKIP 118v (1s) == sanityn test 118w: TCU: repeated deletion with same name for encrypted files ========================================================== 23:22:27 (1779074547) SKIP: sanityn test_118w ldiskfs only test SKIP 118w (2s) == sanityn test 118x: TCU: virtual .Trash for encrypted striped dir ========================================================== 23:22:29 (1779074549) SKIP: sanityn test_118x ldiskfs only test SKIP 118x (1s) == sanityn test 119a: ltrash_purge: verify dry-run mode and --empty mode ========================================================== 23:22:30 (1779074550) SKIP: sanityn test_119a ldiskfs only test SKIP 119a (2s) == sanityn test 119b: ltrash_purge: space-based purging with --auto-ost ========================================================== 23:22:32 (1779074552) SKIP: sanityn test_119b ldiskfs only test SKIP 119b (1s) == sanityn test 119c: ltrash_purge: age-based purging ==== 23:22:33 (1779074553) SKIP: sanityn test_119c ldiskfs only test SKIP 119c (2s) == sanityn test 119d: ltrash_purge: id-based filtering (UID/GID/PROJID) ========================================================== 23:22:35 (1779074555) SKIP: sanityn test_119d ldiskfs only test SKIP 119d (1s) == sanityn test 119e: ltrash_purge: verify --conf option ========================================================== 23:22:36 (1779074556) SKIP: sanityn test_119e ldiskfs only test SKIP 119e (2s) == sanityn test 119f: Run ltrash_purge on subdir mount === 23:22:38 (1779074558) SKIP: sanityn test_119f ldiskfs only test SKIP 119f (1s) == sanityn test 200: service remains healthy while able to process request ========================================================== 23:22:40 (1779074560) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=19 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.097 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0965 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1117 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1033 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1001 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.092 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1031 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0935 s, 0.2 kB/s1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.09 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1549 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1484 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1481 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1959 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1479 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1896 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1474 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1812 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1543 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.1973 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2134 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2449 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2173 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2836 s, 0.1 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 60.268 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2478 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2855 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2265 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3024 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3074 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3423 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3588 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3467 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3102 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3655 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3158 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3585 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 100.392 s, 0.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 100.375 s, 0.0 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg230-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff923242aae000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff923242aae000.ost_server_uuid in IDLE state after 0 sec PASS 200 (108s) cleanup: ====================================================== == sanityn test complete, duration 6241 sec ============== 23:24:28 (1779074668) === sanityn: start cleanup 23:24:28 (1779074668) === Stopping clients: oleg230-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg230-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 23:25:12 (1779074712) ===