-----============= acceptance-small: sanityn ============----- Wed Mar 4 21:39:45 EST 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg111-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a 102 skipping tests SLOW=no: 33a === sanityn: start setup 21:39:54 (1772678394) === Starting client oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 Mount client oleg111-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 Started clients oleg111-client.virtnet: 192.168.201.111@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg111-client.virtnet: executing check_config_client /mnt/lustre oleg111-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg111-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9c1807b0a800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9c180948d800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c1807b0a800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c180948d800.idle_timeout=debug disable quota as required oleg111-server: oleg111-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 21:40:12 (1772678412) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157395 s, 66.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7555] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 21:40:13 (1772678413) PASS 0 (4s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 21:40:17 (1772678417) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (4s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 21:40:22 (1772678422) -rw-r--r-- 1 root root 0 Mar 4 21:40 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (5s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 21:40:26 (1772678426) -rw-r--r-- 1 root root 0 Mar 4 21:40 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (4s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 21:40:30 (1772678430) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (5s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 21:40:35 (1772678435) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (4s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 21:40:40 (1772678440) total 1 -rwxrwxrwx 1 root root 0 Mar 4 21:40 f2a -rwxrwxrwx 1 root root 0 Mar 4 21:40 f2b total 1 -rwxrwxrwx 1 root root 0 Mar 4 21:40 f2a -rwxrwxrwx 1 root root 0 Mar 4 21:40 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.00856099 s, 59.8 kB/s PASS 2e (4s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 21:40:44 (1772678444) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (2s) == sanityn test 2g: check blocks update on sync write ==== 21:40:46 (1772678446) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.256258 s, 8.2 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (5s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 21:40:51 (1772678451) PASS 3 (4s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 21:40:56 (1772678456) PASS 4 (6s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 21:41:01 (1772678461) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (5s) == sanityn test 6: remove of open file on other node ============================================================================ 21:41:06 (1772678466) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (5s) == sanityn test 7: remove of open directory on other node ======================================================================= 21:41:11 (1772678471) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (5s) == sanityn test 8: remove of open special file on other node ==================================================================== 21:41:16 (1772678476) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (5s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 21:41:21 (1772678481) PASS 9a (5s) == sanityn test 9b: append to striped sparse file ======== 21:41:26 (1772678486) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212376 s, 49.4 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0274161 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (4s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 21:41:30 (1772678490) 1+0 records in 1+0 records out 1 byte copied, 0.00554419 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0128949 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.008235 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.011182 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0145385 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.014507 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0136928 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0251329 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0196589 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0121416 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00987719 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0189292 s, 0.1 kB/s PASS 10a (6s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 21:41:36 (1772678496) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0181808 s, 169 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0420877 s, 97.3 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000302692 s, 10.1 MB/s PASS 10b (5s) == sanityn test 11: execution of file opened for write should return error ============================================================== 21:41:41 (1772678501) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7555 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 300: /mnt/lustre2/d11/f: Text file busy PASS 11 (5s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 21:41:46 (1772678506) warning: '-runtime' deprecated, use '-t runtime' instead - link 489 (time 1772678518.12 total 10.02 last 48.80) - link 918 (time 1772678528.13 total 20.03 last 42.84) - link 1380 (time 1772678538.15 total 30.05 last 46.14) - link 1816 (time 1772678548.16 total 40.06 last 43.55) - link 2241 (time 1772678558.17 total 50.07 last 42.44) - link 2683 (time 1772678568.19 total 60.09 last 44.13) - link 3136 (time 1772678578.19 total 70.09 last 45.29) - link 3671 (time 1772678588.22 total 80.12 last 53.36) - link 4202 (time 1772678598.22 total 90.12 last 53.07) total: 4720 link in 100.00 seconds: 47.20 ops/second using seed 712112222 running for 100 seconds - stat 6391 (time 1772678520 ; total 11 ; last 11) - stat 10000 (time 1772678526 ; total 17 ; last 6) - stat 16567 (time 1772678537 ; total 28 ; last 11) - stat 20000 (time 1772678542 ; total 33 ; last 5) - stat 25387 (time 1772678553 ; total 44 ; last 11) - stat 30000 (time 1772678563 ; total 54 ; last 10) - stat 33000 (time 1772678574 ; total 65 ; last 11) - stat 36291 (time 1772678585 ; total 76 ; last 11) - stat 39572 (time 1772678596 ; total 87 ; last 11) - stat 40000 (time 1772678597 ; total 88 ; last 1) - stat 43313 (time 1772678608 ; total 99 ; last 11) total: 43922 stats in 100 seconds: 439.220001 stats/second - unlinked 0 (time 1772678559 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile4720) error: No such file or directory total: 4719 unlinks in 92 seconds: 51.293480 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21334) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21337) - No such process PASS 12 (149s) == sanityn test 13: test directory page revocation ======= 21:44:15 (1772678655) f13 f13 PASS 13 (6s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 21:44:21 (1772678661) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7555 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 335: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 22522 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (4s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 21:44:26 (1772678666) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 23102 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (5s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 21:44:31 (1772678671) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 350: 23682 Terminated $DIR1/$tdir/sleep 60 PASS 14b (4s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 21:44:35 (1772678675) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 362: 24264 Terminated $DIR1/$tdir/sleep 60 PASS 14c (5s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 21:44:40 (1772678680) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 373: 24846 Terminated $DIR1/$tdir/sleep 60 PASS 14d (6s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 21:44:46 (1772678686) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7528448 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7528448KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (37s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 21:45:23 (1772678723) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.235469 s, 35.6 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.119358 s, 70.3 MB/s Chance of close/open is 1 in 50 Seed set to 8724 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x101a1 000004[0] 1772678725.291614 trunc from 00000000 to 0x0101a0 (0x101a1 bytes) 000005[1] 1772678725.350519 read 0x00fe7a thru 0x0101a0 (0x327 bytes) 000006[0] 1772678725.376965 write 0x9875d2 thru 0x98bab9 (0x44e8 bytes) 000007[1] 1772678725.396315 mapwrite 0x9ae5be thru 0x9bd243 (0xec86 bytes) 000010[0] 1772678725.489351 mapread 0x0ea828 thru 0x0eea95 (0x426e bytes) 000015[0] 1772678725.557432 write 0x9e662e thru 0x9f1063 (0xaa36 bytes) 000018[1] 1772678725.602512 mapread 0x3ed1a9 thru 0x3edd41 (0xb99 bytes) 000019[0] 1772678725.627745 write 0x0b3787 thru 0x0b67b9 (0x3033 bytes) 000020[0] 1772678725.657519 mapread 0x6cc798 thru 0x6d1cbf (0x5528 bytes) 000023[0] 1772678725.721201 read 0x1e8408 thru 0x1f2a76 (0xa66f bytes) 000024[0] 1772678725.746593 mapread 0x145da6 thru 0x150deb (0xb046 bytes) 000025[0] 1772678725.784792 mapwrite 0x9b9794 thru 0x9c2c2c (0x9499 bytes) truncating to largest ever: 0x5b0516 000026[0] 1772678725.853630 trunc from 0x9f1064 to 0x5b0515 (0x440b4e bytes) 000027[0] 1772678725.910643 mapwrite 0x549ae6 thru 0x54fa9f (0x5fba bytes) 000029[0] 1772678726.011427 read 0x2d68e6 thru 0x2da17f (0x389a bytes) 000030[1] 1772678726.020840 read 0x1d6c15 thru 0x1dfd9c (0x9188 bytes) 000031[0] 1772678726.043280 write 0x393d82 thru 0x39a8a4 (0x6b23 bytes) 000033[0] 1772678726.055357 mapread 0x460018 thru 0x46d536 (0xd51f bytes) 000038[0] 1772678726.092993 mapread 0x18cbd4 thru 0x19a330 (0xd75d bytes) 000041[1] 1772678726.136502 mapread 0x57d2d0 thru 0x5829af (0x56e0 bytes) 000043[0] 1772678726.237171 read 0x29b56e thru 0x2a7e40 (0xc8d3 bytes) truncating to largest ever: 0x967b33 000047[1] 1772678726.339589 trunc from 0x5b0516 to 0x967b32 (0x3b761d bytes) 000048[1] 1772678726.409318 read 0x6693c5 thru 0x677e27 (0xea63 bytes) 000049[1] 1772678726.456650 read 0x3668db thru 0x37633b (0xfa61 bytes) 000051[0] 1772678726.505678 trunc from 0x967b33 to 0x60221b (0x365917 bytes) 000052[0] 1772678726.564369 read 0x0bc832 thru 0x0bfaeb (0x32ba bytes) 000053[0] 1772678726.572578 mapwrite 0x7b3129 thru 0x7c23ab (0xf283 bytes) 000054[0] 1772678726.694794 trunc from 0x7c23ac to 0x8d225f (0x10feb4 bytes) 000055[1] 1772678726.751621 mapwrite 0x0e38b2 thru 0x0ea776 (0x6ec5 bytes) 000057[1] 1772678726.894497 trunc from 0x8d2260 to 0x73ef9a (0x1932c5 bytes) 000060[1] 1772678726.918749 mapread 0x6ea21d thru 0x6f628e (0xc072 bytes) 000061[1] 1772678726.949876 mapread 0x46b2ab thru 0x473887 (0x85dd bytes) 000062[0] 1772678726.994494 trunc from 0x73ef9b to 0x10acb6 (0x6342e4 bytes) 000064[0] 1772678727.047252 read 0x10808d thru 0x108daf (0xd23 bytes) 000065[0] 1772678727.081144 write 0x6954b8 thru 0x69b58e (0x60d7 bytes) 000066[0] 1772678727.087623 write 0x85075e thru 0x85efb2 (0xe855 bytes) 000068[0] 1772678727.111478 mapwrite 0x6ed9ba thru 0x6f9929 (0xbf70 bytes) 000070[0] 1772678727.283896 write 0x36f481 thru 0x3727df (0x335f bytes) 000071[0] 1772678727.300643 trunc from 0x85efb3 to 0x36b4ba (0x4f3af8 bytes) 000072[0] 1772678727.354578 mapread 0x0b5e19 thru 0x0c5347 (0xf52f bytes) 000074[0] 1772678727.372695 mapread 0x2f5b2e thru 0x304711 (0xebe4 bytes) 000075[0] 1772678727.373838 trunc from 0x36b4bb to 0x2e5f8f (0x8552b bytes) 000076[1] 1772678727.405538 read 0x25ba65 thru 0x265f1b (0xa4b7 bytes) 000077[0] 1772678727.479717 trunc from 0x2e5f90 to 0x27075e (0x75831 bytes) 000078[0] 1772678727.519727 mapread 0x1ca7ad thru 0x1d35c0 (0x8e14 bytes) 000081[1] 1772678727.626644 mapread 0x0a5fba thru 0x0aebfe (0x8c45 bytes) 000082[0] 1772678727.722152 read 0x1a81d8 thru 0x1af5df (0x7408 bytes) 000084[0] 1772678727.767225 mapwrite 0x37e6d6 thru 0x384f59 (0x6884 bytes) 000085[1] 1772678727.932028 write 0x253a7d thru 0x2582e2 (0x4866 bytes) 000086[0] 1772678727.990626 trunc from 0x384f5a to 0x3f5f2b (0x70fd2 bytes) 000087[1] 1772678728.052844 mapwrite 0x9900c8 thru 0x99c5c4 (0xc4fd bytes) 000089[0] 1772678728.303001 read 0x432446 thru 0x4414d8 (0xf093 bytes) 000090[1] 1772678728.347625 trunc from 0x99c5c5 to 0x88bf64 (0x110660 bytes) 000091[0] 1772678728.383972 mapread 0x2285c8 thru 0x23467d (0xc0b6 bytes) 000094[1] 1772678728.454222 mapread 0x22c96f thru 0x2320ad (0x573f bytes) 000096[1] 1772678728.524555 mapwrite 0x0972cb thru 0x0a4a13 (0xd749 bytes) 000098[0] 1772678728.696423 trunc from 0x88bf65 to 0x7bced7 (0xcf08d bytes) 000099[1] 1772678728.726428 trunc from 0x7bced8 to 0x1e8eb5 (0x5d4022 bytes) 000101[0] 1772678728.787152 trunc from 0x1e8eb6 to 0x7c496b (0x5dbab6 bytes) 000104[1] 1772678728.894540 mapwrite 0x609bce thru 0x611bd3 (0x8006 bytes) 000105[0] 1772678729.031532 mapread 0x694a6d thru 0x6a45a2 (0xfb36 bytes) 000107[1] 1772678729.128813 write 0x7000f1 thru 0x7080cc (0x7fdc bytes) 000109[0] 1772678729.203159 read 0x4dfa33 thru 0x4e942c (0x99fa bytes) 000110[1] 1772678729.238268 write 0x420c76 thru 0x42fe81 (0xf20c bytes) 000118[0] 1772678729.436566 write 0x61a951 thru 0x627489 (0xcb39 bytes) 000121[0] 1772678729.570621 mapwrite 0x0d22e8 thru 0x0e1313 (0xf02c bytes) 000122[0] 1772678729.673273 mapread 0x4551a3 thru 0x45dc9d (0x8afb bytes) 000123[1] 1772678729.694802 mapwrite 0x43bd82 thru 0x4483c2 (0xc641 bytes) 000126[1] 1772678729.826979 trunc from 0x7c496c to 0x416cef (0x3adc7c bytes) 000127[0] 1772678729.877764 read 0x1446c2 thru 0x149f90 (0x58cf bytes) 000128[0] 1772678729.912107 mapwrite 0x68e450 thru 0x6980d2 (0x9c83 bytes) 000129[1] 1772678730.033991 trunc from 0x6980d3 to 0x4eb354 (0x1acd7e bytes) 000131[1] 1772678730.100715 read 0x4a0968 thru 0x4a68d7 (0x5f70 bytes) 000134[0] 1772678730.127135 trunc from 0x4eb355 to 0x81aae2 (0x32f78e bytes) 000135[1] 1772678730.195886 mapread 0x4a0707 thru 0x4a28c1 (0x21bb bytes) 000136[0] 1772678730.231468 write 0x84ca4a thru 0x84ffaf (0x3566 bytes) 000137[1] 1772678730.232621 mapread 0x51985d thru 0x51bb1e (0x22c2 bytes) 000138[1] 1772678730.252710 mapread 0x2ff29d thru 0x30865b (0x93bf bytes) 000139[0] 1772678730.298491 mapwrite 0x5a40aa thru 0x5b169f (0xd5f6 bytes) 000140[0] 1772678730.363271 mapwrite 0x5b0a88 thru 0x5bbcd3 (0xb24c bytes) 000141[0] 1772678730.431656 mapread 0x2b7949 thru 0x2c1fcf (0xa687 bytes) 000145[0] 1772678730.472690 mapwrite 0x29b8be thru 0x2a944b (0xdb8e bytes) 000147[1] 1772678730.574230 write 0x4709a5 thru 0x47217e (0x17da bytes) 000149[1] 1772678730.614805 read 0x7fe4c7 thru 0x80c01b (0xdb55 bytes) 000150[0] 1772678730.643518 write 0x17997b thru 0x17d50d (0x3b93 bytes) 000151[1] 1772678730.664567 read 0x78eeb7 thru 0x79d8d7 (0xea21 bytes) 000154[0] 1772678730.736062 mapread 0x120126 thru 0x12b206 (0xb0e1 bytes) 000157[0] 1772678730.824489 mapwrite 0x9af3a1 thru 0x9beb19 (0xf779 bytes) 000158[0] 1772678730.978028 read 0x4cac81 thru 0x4cbc90 (0x1010 bytes) 000160[0] 1772678731.041658 mapwrite 0x843329 thru 0x84cb80 (0x9858 bytes) 000162[1] 1772678731.196667 mapread 0x8980f4 thru 0x89e6c3 (0x65d0 bytes) 000163[1] 1772678731.239638 write 0x25dc23 thru 0x26c37b (0xe759 bytes) 000164[0] 1772678731.272117 mapread 0x063313 thru 0x07093d (0xd62b bytes) 000165[1] 1772678731.316367 mapread 0x912a11 thru 0x91eeb2 (0xc4a2 bytes) 000167[0] 1772678731.370237 write 0x5e8977 thru 0x5f87b9 (0xfe43 bytes) 000169[0] 1772678731.396551 mapread 0x7d46d0 thru 0x7d9f41 (0x5872 bytes) 000170[1] 1772678731.429274 read 0x97c035 thru 0x9897cf (0xd79b bytes) 000171[1] 1772678731.459785 write 0x5a46a6 thru 0x5ae2be (0x9c19 bytes) 000176[1] 1772678731.518380 write 0x8dc08d thru 0x8e4162 (0x80d6 bytes) 000178[0] 1772678731.563921 write 0x2dcd64 thru 0x2e84a5 (0xb742 bytes) 000180[1] 1772678731.651246 mapread 0x0d19ad thru 0x0d2fb5 (0x1609 bytes) 000181[1] 1772678731.706160 mapwrite 0x1c61f8 thru 0x1c8f76 (0x2d7f bytes) 000182[1] 1772678731.797183 write 0x4d2e6c thru 0x4dccef (0x9e84 bytes) 000184[1] 1772678731.818857 mapwrite 0x9f216e thru 0x9fffff (0xde92 bytes) 000186[1] 1772678731.954629 mapwrite 0x56ead5 thru 0x575582 (0x6aae bytes) 000188[1] 1772678732.041480 write 0x3ac404 thru 0x3b24aa (0x60a7 bytes) 000194[1] 1772678732.117817 trunc from 0xa00000 to 0x189fdf (0x876020 bytes) 000195[1] 1772678732.135580 mapread 0x0dd6a5 thru 0x0e1f73 (0x48cf bytes) 000196[0] 1772678732.152150 trunc from 0x189fe0 to 0x51b92d (0x39194e bytes) 000197[0] 1772678732.210808 read 0x09fc74 thru 0x0a74ed (0x787a bytes) 000198[1] 1772678732.221883 trunc from 0x51b92e to 0x109720 (0x41220d bytes) 000199[0] 1772678732.270810 read 0x036a72 thru 0x044681 (0xdc10 bytes) 000200[1] 1772678732.316181 mapwrite 0x5ba44e thru 0x5bf03a (0x4bed bytes) 000202[1] 1772678732.425728 mapwrite 0x42642d thru 0x43137d (0xaf51 bytes) 000205[0] 1772678732.546214 read 0x59905e thru 0x59a46b (0x140e bytes) 000206[1] 1772678732.577887 write 0x5246a0 thru 0x53343d (0xed9e bytes) 000207[1] 1772678732.618235 trunc from 0x5bf03b to 0x200270 (0x3bedca bytes) 000208[0] 1772678732.647811 read 0x06f2f9 thru 0x072392 (0x309a bytes) 000211[1] 1772678732.700991 write 0x4a64d3 thru 0x4a7e26 (0x1954 bytes) 000213[1] 1772678732.722286 mapread 0x11f812 thru 0x12ec1e (0xf40d bytes) 000214[1] 1772678732.753274 write 0x0cd5f5 thru 0x0d0dab (0x37b7 bytes) 000215[0] 1772678732.780269 write 0x654c30 thru 0x65637b (0x174c bytes) 000217[0] 1772678732.819330 mapread 0x5364bc thru 0x53e268 (0x7dad bytes) 000219[0] 1772678732.897538 mapwrite 0x677f11 thru 0x6869f5 (0xeae5 bytes) 000221[0] 1772678733.084888 write 0x7decf2 thru 0x7e4caa (0x5fb9 bytes) 000223[0] 1772678733.104174 mapwrite 0x54a78b thru 0x55289a (0x8110 bytes) 000224[0] 1772678733.152791 mapwrite 0x0ab551 thru 0x0bb306 (0xfdb6 bytes) 000225[0] 1772678733.330333 mapread 0x09f27a thru 0x0a0be5 (0x196c bytes) 000226[1] 1772678733.344153 trunc from 0x7e4cab to 0x58b780 (0x25952a bytes) 000227[1] 1772678733.421119 write 0x75fc3b thru 0x766768 (0x6b2e bytes) 000228[0] 1772678733.428481 mapread 0x4a6f29 thru 0x4ad817 (0x68ef bytes) 000232[0] 1772678733.531487 mapwrite 0x4024fe thru 0x402603 (0x106 bytes) 000234[0] 1772678733.622910 write 0x63fc79 thru 0x64be51 (0xc1d9 bytes) 000236[0] 1772678733.645960 read 0x4f3356 thru 0x4f4dc2 (0x1a6d bytes) 000237[0] 1772678733.664903 mapwrite 0x67b8d4 thru 0x6885a1 (0xccce bytes) 000239[0] 1772678733.760133 mapread 0x705f0c thru 0x70b9f0 (0x5ae5 bytes) 000244[0] 1772678733.877880 trunc from 0x766769 to 0x03dc90 (0x728ad8 bytes) 000245[1] 1772678733.900544 mapwrite 0x95f8e4 thru 0x96db00 (0xe21d bytes) 000248[1] 1772678734.051719 read 0x4160a8 thru 0x41d95d (0x78b6 bytes) 000252[1] 1772678734.098565 mapwrite 0x4d3cd6 thru 0x4d4264 (0x58f bytes) 000253[0] 1772678734.174174 write 0x53dd3d thru 0x546230 (0x84f4 bytes) 000254[1] 1772678734.196408 mapwrite 0x85f0b0 thru 0x86552a (0x647b bytes) 000257[0] 1772678734.347752 read 0x66a37a thru 0x66dfc0 (0x3c47 bytes) 000261[1] 1772678734.426489 read 0x31a0dd thru 0x31bbea (0x1b0e bytes) 000264[1] 1772678734.476158 read 0x038bd0 thru 0x043db6 (0xb1e7 bytes) 000267[1] 1772678734.528924 read 0x6af732 thru 0x6b62c0 (0x6b8f bytes) 000268[0] 1772678734.555089 trunc from 0x96db01 to 0x944ca3 (0x28e5d bytes) 000269[1] 1772678734.601593 mapwrite 0x4cb1d4 thru 0x4cd2cd (0x20fa bytes) 000270[1] 1772678734.678432 mapread 0x419c99 thru 0x42830e (0xe676 bytes) 000273[0] 1772678734.762428 mapwrite 0x30d57b thru 0x31c57c (0xf002 bytes) 000274[1] 1772678734.905256 write 0x727ebe thru 0x7328f8 (0xaa3b bytes) 000276[0] 1772678734.945676 mapread 0x534dbb thru 0x539d2d (0x4f73 bytes) 000278[1] 1772678734.980591 read 0x099c9c thru 0x0a99ca (0xfd2f bytes) 000279[1] 1772678735.001267 write 0x01cde4 thru 0x024485 (0x76a2 bytes) 000282[1] 1772678735.025331 mapwrite 0x927d1c thru 0x92bcd0 (0x3fb5 bytes) 000283[1] 1772678735.084205 mapread 0x77de28 thru 0x7885a0 (0xa779 bytes) 000284[1] 1772678735.105246 trunc from 0x944ca4 to 0x759868 (0x1eb43b bytes) 000286[1] 1772678735.129124 mapwrite 0x1f70dd thru 0x2032db (0xc1ff bytes) 000289[1] 1772678735.261759 trunc from 0x759869 to 0x0ae6ef (0x6ab179 bytes) 000291[1] 1772678735.310682 mapwrite 0x11d10f thru 0x11ebfb (0x1aed bytes) 000295[1] 1772678735.463397 write 0x95c162 thru 0x95e3e1 (0x2280 bytes) 000302[0] 1772678735.498353 read 0x4933e4 thru 0x49a666 (0x7283 bytes) 000304[1] 1772678735.578487 read 0x645f8a thru 0x64697b (0x9f2 bytes) 000305[0] 1772678735.599972 mapread 0x2d1f43 thru 0x2d9ba9 (0x7c67 bytes) 000306[0] 1772678735.637600 read 0x44d696 thru 0x45853d (0xaea8 bytes) 000307[0] 1772678735.644206 trunc from 0x95e3e2 to 0x4e4932 (0x479aaf bytes) 000312[1] 1772678735.716688 read 0x31d62f thru 0x31e82d (0x11ff bytes) 000315[1] 1772678735.740767 trunc from 0x4e4933 to 0x0a2aea (0x441e48 bytes) 000320[1] 1772678735.788955 mapread 0x064c17 thru 0x072cae (0xe098 bytes) 000322[1] 1772678735.851135 write 0x10db33 thru 0x113319 (0x57e7 bytes) 000324[1] 1772678735.901840 read 0x0a83bf thru 0x0a88f4 (0x536 bytes) 000325[1] 1772678735.908589 mapread 0x0bd2a7 thru 0x0bd813 (0x56d bytes) 000327[0] 1772678735.918351 trunc from 0x11331a to 0x0ed778 (0x25ba1 bytes) 000331[1] 1772678736.028796 read 0x0c9682 thru 0x0d571a (0xc099 bytes) 000332[1] 1772678736.045245 mapwrite 0x129787 thru 0x1377de (0xe058 bytes) 000333[1] 1772678736.169934 read 0x021e3a thru 0x030a30 (0xebf7 bytes) 000336[0] 1772678736.202690 mapread 0x0201bb thru 0x02c2a6 (0xc0ec bytes) 000340[0] 1772678736.277248 mapread 0x0a3947 thru 0x0a74f3 (0x3bad bytes) 000343[0] 1772678736.342836 mapread 0x0ddf90 thru 0x0ea7bb (0xc82c bytes) 000350[0] 1772678736.488916 mapwrite 0x1176a4 thru 0x1191f9 (0x1b56 bytes) 000354[1] 1772678736.628565 mapread 0x030a4d thru 0x032b85 (0x2139 bytes) 000359[0] 1772678736.690521 read 0x0fa6a9 thru 0x0fc4fe (0x1e56 bytes) 000360[1] 1772678736.700073 mapread 0x02dc0b thru 0x0313eb (0x37e1 bytes) 000361[0] 1772678736.722107 mapread 0x04f1bc thru 0x05ef23 (0xfd68 bytes) 000364[0] 1772678736.741459 mapwrite 0x43cc84 thru 0x4493f7 (0xc774 bytes) 000366[0] 1772678736.856508 mapwrite 0x1b9f12 thru 0x1ba442 (0x531 bytes) 000367[0] 1772678736.896398 read 0x1ae655 thru 0x1b0b54 (0x2500 bytes) 000368[1] 1772678736.917425 read 0x26521f thru 0x266dd3 (0x1bb5 bytes) 000369[1] 1772678736.941742 mapwrite 0x1fe371 thru 0x205c0f (0x789f bytes) 000370[1] 1772678737.036354 write 0x160adb thru 0x16d340 (0xc866 bytes) 000371[0] 1772678737.077282 write 0x65c3a0 thru 0x66345f (0x70c0 bytes) 000373[0] 1772678737.102361 trunc from 0x663460 to 0x01fbae (0x6438b1 bytes) 000375[1] 1772678737.140936 trunc from 0x01fbaf to 0x1b9407 (0x199859 bytes) 000377[1] 1772678737.192422 mapwrite 0x463ab0 thru 0x46808c (0x45dd bytes) 000379[0] 1772678737.295293 trunc from 0x46808d to 0x6e1b85 (0x279af9 bytes) 000382[1] 1772678737.375190 write 0x23d31f thru 0x240dab (0x3a8d bytes) 000386[1] 1772678737.421970 mapread 0x2ee50f thru 0x2f0684 (0x2176 bytes) 000387[1] 1772678737.449164 mapwrite 0x4306ad thru 0x432284 (0x1bd8 bytes) 000390[0] 1772678737.556349 write 0x86e784 thru 0x87bafc (0xd379 bytes) 000391[0] 1772678737.591455 write 0x11fd6b thru 0x121301 (0x1597 bytes) 000392[1] 1772678737.604026 mapwrite 0x23f0a7 thru 0x2413d6 (0x2330 bytes) 000393[1] 1772678737.656624 write 0x61d60b thru 0x62b8ec (0xe2e2 bytes) 000394[1] 1772678737.670658 trunc from 0x87bafd to 0x541785 (0x33a377 bytes) 000400[1] 1772678737.720673 mapwrite 0x94ea8a thru 0x95ab0f (0xc086 bytes) 000401[1] 1772678737.832566 write 0x1e9249 thru 0x1f53ba (0xc172 bytes) 000402[0] 1772678737.858268 mapread 0x54d788 thru 0x54e284 (0xafd bytes) 000404[0] 1772678737.894389 mapwrite 0x9e11a1 thru 0x9e3c0a (0x2a6a bytes) 000407[1] 1772678738.035373 mapread 0x1b6b24 thru 0x1bcfe0 (0x64bd bytes) 000408[1] 1772678738.055789 write 0x50d4e6 thru 0x51c1af (0xecca bytes) 000409[1] 1772678738.087089 trunc from 0x9e3c0b to 0x6588ed (0x38b31d bytes) 000411[0] 1772678738.138303 mapread 0x59838d thru 0x59aef2 (0x2b66 bytes) 000412[0] 1772678738.184072 trunc from 0x6588ee to 0x912f82 (0x2ba695 bytes) 000415[0] 1772678738.224386 mapwrite 0x96961d thru 0x977038 (0xda1c bytes) 000416[0] 1772678738.333908 read 0x09ddfe thru 0x0a5216 (0x7419 bytes) 000419[0] 1772678738.377189 write 0x976038 thru 0x977a6b (0x1a34 bytes) 000421[1] 1772678738.380031 write 0x8ad49e thru 0x8afbaf (0x2712 bytes) 000423[1] 1772678738.440549 mapread 0x372212 thru 0x3754e3 (0x32d2 bytes) 000425[1] 1772678738.487624 write 0x995585 thru 0x9960ea (0xb66 bytes) 000426[0] 1772678738.503289 write 0x43b0b8 thru 0x44299f (0x78e8 bytes) 000432[0] 1772678738.581347 mapwrite 0x738490 thru 0x7392c1 (0xe32 bytes) 000434[1] 1772678738.673014 mapwrite 0x35cf96 thru 0x36ac49 (0xdcb4 bytes) 000435[1] 1772678738.758138 write 0x7f86ce thru 0x808030 (0xf963 bytes) 000436[0] 1772678738.783805 read 0x778d6c thru 0x77ffdf (0x7274 bytes) 000437[1] 1772678738.807310 write 0x1a1888 thru 0x1a4027 (0x27a0 bytes) 000444[1] 1772678738.917619 read 0x4af39d thru 0x4b23c5 (0x3029 bytes) 000446[1] 1772678738.956529 write 0x23dc60 thru 0x242d98 (0x5139 bytes) 000447[1] 1772678738.967866 mapwrite 0x732cc7 thru 0x735f6d (0x32a7 bytes) 000455[0] 1772678739.070265 trunc from 0x9960eb to 0x5724e6 (0x423c04 bytes) 000456[0] 1772678739.118588 trunc from 0x5724e7 to 0x1b9588 (0x3b8f5e bytes) 000457[1] 1772678739.134998 write 0x3d4b99 thru 0x3e330e (0xe776 bytes) 000463[0] 1772678739.275562 read 0x1aae4a thru 0x1af074 (0x422b bytes) 000464[0] 1772678739.323201 mapread 0x3ced2d thru 0x3cf751 (0xa25 bytes) 000465[0] 1772678739.341589 trunc from 0x3e330f to 0x48246f (0x9f161 bytes) 000467[0] 1772678739.406629 mapwrite 0x067f57 thru 0x06f1a0 (0x724a bytes) 000469[0] 1772678739.512469 read 0x29c2da thru 0x2a8716 (0xc43d bytes) 000470[0] 1772678739.521362 mapread 0x0cf3b0 thru 0x0d4017 (0x4c68 bytes) 000471[0] 1772678739.550776 mapwrite 0x965992 thru 0x96ce66 (0x74d5 bytes) 000472[0] 1772678739.651939 trunc from 0x96ce67 to 0x7e5276 (0x187bf0 bytes) 000475[1] 1772678739.691584 mapwrite 0x5a90d9 thru 0x5b55b0 (0xc4d8 bytes) 000478[1] 1772678739.806785 mapread 0x1d3dd9 thru 0x1defd3 (0xb1fb bytes) 000479[0] 1772678739.866788 read 0x151f2d thru 0x1579a8 (0x5a7c bytes) 000480[1] 1772678739.899125 mapread 0x4a0056 thru 0x4a2f9e (0x2f49 bytes) 000481[0] 1772678739.952179 read 0x1c8c84 thru 0x1cd7a0 (0x4b1d bytes) 000483[1] 1772678739.990633 mapread 0x5d7732 thru 0x5dc43a (0x4d09 bytes) 000484[0] 1772678740.011838 read 0x001b03 thru 0x00a62f (0x8b2d bytes) 000485[0] 1772678740.019468 mapread 0x0c9d0e thru 0x0ca386 (0x679 bytes) 000488[0] 1772678740.054398 mapread 0x331fe2 thru 0x33c0df (0xa0fe bytes) 000489[0] 1772678740.118727 trunc from 0x7e5277 to 0x3378b0 (0x4ad9c6 bytes) 000490[1] 1772678740.172458 read 0x0c6943 thru 0x0d557c (0xec3a bytes) 000491[1] 1772678740.227789 read 0x0827a4 thru 0x08d7be (0xb01b bytes) 000494[1] 1772678740.324323 read 0x0c9d3b thru 0x0d4bd5 (0xae9b bytes) 000495[1] 1772678740.351329 write 0x3e93e1 thru 0x3f4c96 (0xb8b6 bytes) 000496[0] 1772678740.380574 write 0x7a3bee thru 0x7a9490 (0x58a3 bytes) 000497[1] 1772678740.398836 write 0x9fc48f thru 0x9fffff (0x3b71 bytes) 000498[0] 1772678740.423405 write 0x09b023 thru 0x0a0e81 (0x5e5f bytes) 000500[1] 1772678740.486410 read 0x8347d7 thru 0x834ee3 (0x70d bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 8740 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000010[1] 1772678741.234061 write 0x229000 thru 0x23306a (0xa06b bytes) 000011[0] 1772678741.237030 mapwrite 0x367000 thru 0x36793f (0x940 bytes) 000013[0] 1772678741.321745 write_OD 0x4ed000 thru 0x4fafff (0xe000 bytes) truncating to largest ever: 0x241dd3 000015[1] 1772678741.379008 trunc from 0x4fb000 to 0x241dd2 (0x2b922d bytes) truncating to largest ever: 0x6b921c 000016[0] 1772678741.404479 trunc from 0x241dd3 to 0x6b921b (0x477449 bytes) 000017[1] 1772678741.465872 mapread 0x5c6000 thru 0x5d2db4 (0xcdb5 bytes) 000018[1] 1772678741.526944 write 0x22d000 thru 0x22d268 (0x269 bytes) 000019[1] 1772678741.541797 read 0x09f000 thru 0x0a44a3 (0x54a4 bytes) 000023[1] 1772678741.589817 mapread 0x235000 thru 0x24341d (0xe41e bytes) 000024[1] 1772678741.628659 write 0x9f2000 thru 0x9f90ad (0x70ae bytes) 000025[0] 1772678741.632191 mapread 0x86b000 thru 0x873c06 (0x8c07 bytes) 000026[1] 1772678741.691467 read 0x676000 thru 0x678c74 (0x2c75 bytes) 000028[0] 1772678741.698366 read_OD 0x9cb000 thru 0x9cdfff (0x3000 bytes) 000029[1] 1772678741.703969 read 0x2df000 thru 0x2eae5b (0xbe5c bytes) 000031[0] 1772678741.714283 read_OD 0x3a6000 thru 0x3a6fff (0x1000 bytes) 000033[1] 1772678741.720755 write 0x0bf000 thru 0x0cb573 (0xc574 bytes) 000034[0] 1772678741.757172 mapwrite 0x0d0000 thru 0x0d71a7 (0x71a8 bytes) 000035[0] 1772678741.827206 mapwrite 0x3f4000 thru 0x3f7907 (0x3908 bytes) 000036[1] 1772678741.889856 mapread 0x1c0000 thru 0x1cbbfa (0xbbfb bytes) 000039[1] 1772678741.940681 mapwrite 0x5e9000 thru 0x5ecdde (0x3ddf bytes) 000040[1] 1772678741.993320 read 0x819000 thru 0x8289b4 (0xf9b5 bytes) 000041[1] 1772678742.004317 read 0x196000 thru 0x1a430c (0xe30d bytes) 000042[1] 1772678742.029397 mapread 0x0d6000 thru 0x0d9d4a (0x3d4b bytes) 000043[1] 1772678742.070752 write 0x91e000 thru 0x920c1e (0x2c1f bytes) 000050[1] 1772678742.131216 mapread 0x252000 thru 0x258305 (0x6306 bytes) 000052[0] 1772678742.153710 write_OD 0x3c7000 thru 0x3cbfff (0x5000 bytes) 000053[1] 1772678742.200381 read 0x57c000 thru 0x582b38 (0x6b39 bytes) 000056[0] 1772678742.219794 mapwrite 0x122000 thru 0x1261db (0x41dc bytes) 000058[0] 1772678742.304861 mapread 0x45d000 thru 0x461325 (0x4326 bytes) 000059[0] 1772678742.332575 write_OD 0x8f9000 thru 0x8fefff (0x6000 bytes) 000061[0] 1772678742.380391 mapwrite 0x0a7000 thru 0x0b1efc (0xaefd bytes) 000065[0] 1772678742.485963 trunc from 0x9f90ae to 0x5e4bb5 (0x4144f8 bytes) 000067[0] 1772678742.501883 mapread 0x199000 thru 0x1a11ef (0x81f0 bytes) 000070[0] 1772678742.564514 trunc from 0x5e4bb6 to 0x30a614 (0x2da5a1 bytes) 000072[0] 1772678742.595224 trunc from 0x30a615 to 0x08fd2d (0x27a8e7 bytes) 000073[0] 1772678742.620078 read_OD 0x039000 thru 0x03cfff (0x4000 bytes) truncating to largest ever: 0x746199 000075[1] 1772678742.634655 trunc from 0x08fd2e to 0x746198 (0x6b646b bytes) 000077[0] 1772678742.671171 read_OD 0x40b000 thru 0x419fff (0xf000 bytes) 000078[1] 1772678742.698944 read 0x234000 thru 0x23bf86 (0x7f87 bytes) truncating to largest ever: 0x8c3603 000079[1] 1772678742.708693 trunc from 0x746199 to 0x8c3602 (0x17d46a bytes) 000080[1] 1772678742.742588 mapread 0x58a000 thru 0x593d9f (0x9da0 bytes) 000082[1] 1772678742.768189 read 0x716000 thru 0x718e5e (0x2e5f bytes) 000083[0] 1772678742.781228 write_OD 0x0df000 thru 0x0e5fff (0x7000 bytes) 000087[0] 1772678742.821602 mapwrite 0x989000 thru 0x994919 (0xb91a bytes) 000088[1] 1772678742.948382 mapwrite 0x56c000 thru 0x573cfe (0x7cff bytes) 000089[1] 1772678743.027990 mapread 0x23f000 thru 0x24ed34 (0xfd35 bytes) 000090[1] 1772678743.051822 read 0x8ff000 thru 0x8ffdac (0xdad bytes) 000091[0] 1772678743.055710 trunc from 0x99491a to 0x8a44b1 (0xf0468 bytes) 000095[1] 1772678743.112404 mapwrite 0x198000 thru 0x19ff61 (0x7f62 bytes) 000096[1] 1772678743.179211 read 0x280000 thru 0x2884b9 (0x84ba bytes) 000098[1] 1772678743.222475 write 0x7f3000 thru 0x800cb2 (0xdcb3 bytes) 000099[0] 1772678743.245026 read_OD 0x17d000 thru 0x189fff (0xd000 bytes) 000100[0] 1772678743.278175 read_OD 0x58d000 thru 0x58dfff (0x1000 bytes) 000101[0] 1772678743.289313 mapread 0x856000 thru 0x85df51 (0x7f52 bytes) 000102[0] 1772678743.309730 read_OD 0x141000 thru 0x141fff (0x1000 bytes) 000105[1] 1772678743.316943 trunc from 0x8a44b2 to 0x8ba3b0 (0x15eff bytes) 000106[1] 1772678743.341682 read 0x7fd000 thru 0x80a054 (0xd055 bytes) 000107[1] 1772678743.350285 mapwrite 0x631000 thru 0x63b6d5 (0xa6d6 bytes) 000108[1] 1772678743.424126 trunc from 0x8ba3b1 to 0x562cda (0x3576d6 bytes) 000110[1] 1772678743.436774 read 0x43a000 thru 0x448feb (0xefec bytes) 000112[1] 1772678743.487300 read 0x155000 thru 0x15f7f6 (0xa7f7 bytes) 000118[0] 1772678743.516573 trunc from 0x562cdb to 0x664f8c (0x1022b2 bytes) 000120[0] 1772678743.580662 mapwrite 0x0f9000 thru 0x0ffa05 (0x6a06 bytes) 000122[1] 1772678743.697754 trunc from 0x664f8d to 0x201f9b (0x462ff1 bytes) 000123[0] 1772678743.742615 trunc from 0x201f9c to 0x5bcd3c (0x3bada1 bytes) 000124[1] 1772678743.795735 trunc from 0x5bcd3d to 0x602fc5 (0x46289 bytes) 000126[1] 1772678743.832316 write 0x328000 thru 0x32a020 (0x2021 bytes) 000129[0] 1772678743.872189 write_OD 0x9e2000 thru 0x9e6fff (0x5000 bytes) 000130[0] 1772678743.931548 trunc from 0x9e7000 to 0x83cd95 (0x1aa26a bytes) 000131[1] 1772678743.984725 write 0x47a000 thru 0x4827e3 (0x87e4 bytes) 000132[0] 1772678744.010105 trunc from 0x83cd96 to 0x5ac518 (0x29087d bytes) 000133[0] 1772678744.063639 write_OD 0x0f6000 thru 0x0fffff (0xa000 bytes) 000135[0] 1772678744.154734 write_OD 0x420000 thru 0x421fff (0x2000 bytes) 000136[1] 1772678744.192673 read 0x1d6000 thru 0x1db757 (0x5758 bytes) 000141[0] 1772678744.206906 mapwrite 0x234000 thru 0x238735 (0x4736 bytes) 000142[0] 1772678744.345700 mapwrite 0x31b000 thru 0x325d71 (0xad72 bytes) 000143[1] 1772678744.442624 mapread 0x42b000 thru 0x430d1a (0x5d1b bytes) 000147[0] 1772678744.497878 read_OD 0x1b4000 thru 0x1bbfff (0x8000 bytes) 000148[0] 1772678744.514301 mapread 0x22b000 thru 0x23553b (0xa53c bytes) 000149[0] 1772678744.539375 mapread 0x269000 thru 0x274b10 (0xbb11 bytes) 000152[1] 1772678744.564523 trunc from 0x5ac519 to 0x06576c (0x546dac bytes) 000158[0] 1772678744.638956 mapread 0x027000 thru 0x0326ab (0xb6ac bytes) 000159[1] 1772678744.690097 write 0x76b000 thru 0x76b5db (0x5dc bytes) 000160[1] 1772678744.708291 read 0x4f7000 thru 0x504a88 (0xda89 bytes) 000161[0] 1772678744.718275 mapread 0x55e000 thru 0x56328a (0x528b bytes) 000163[0] 1772678744.787799 mapwrite 0x226000 thru 0x227d6d (0x1d6e bytes) 000167[1] 1772678744.847749 mapread 0x0a3000 thru 0x0a403e (0x103f bytes) 000169[1] 1772678744.874420 mapread 0x4d8000 thru 0x4e7495 (0xf496 bytes) 000170[1] 1772678744.930766 trunc from 0x76b5dc to 0x74521b (0x263c0 bytes) 000171[0] 1772678744.976573 trunc from 0x74521c to 0x069455 (0x6dbdc6 bytes) 000174[0] 1772678745.041170 read_OD 0x041000 thru 0x043fff (0x3000 bytes) 000175[1] 1772678745.064819 mapwrite 0x554000 thru 0x557617 (0x3618 bytes) 000176[0] 1772678745.164258 mapwrite 0x58e000 thru 0x59543f (0x7440 bytes) 000178[0] 1772678745.342486 mapread 0x268000 thru 0x26a371 (0x2372 bytes) 000180[1] 1772678745.343949 write 0x204000 thru 0x20958c (0x558d bytes) 000187[0] 1772678745.384529 mapwrite 0x068000 thru 0x075c2f (0xdc30 bytes) 000191[0] 1772678745.565106 mapwrite 0x301000 thru 0x302cfa (0x1cfb bytes) 000192[0] 1772678745.629102 mapread 0x23f000 thru 0x2469bf (0x79c0 bytes) 000193[1] 1772678745.669545 mapwrite 0x12a000 thru 0x12d8cd (0x38ce bytes) 000194[1] 1772678745.749284 trunc from 0x595440 to 0x609594 (0x74155 bytes) 000196[0] 1772678745.767728 read_OD 0x291000 thru 0x293fff (0x3000 bytes) 000197[1] 1772678745.789247 read 0x5fc000 thru 0x603b34 (0x7b35 bytes) 000199[1] 1772678745.799233 read 0x3f6000 thru 0x3fb578 (0x5579 bytes) 000200[0] 1772678745.807148 mapwrite 0x65f000 thru 0x662383 (0x3384 bytes) 000201[0] 1772678745.902961 mapread 0x1ec000 thru 0x1f8930 (0xc931 bytes) 000202[1] 1772678745.937519 read 0x4cf000 thru 0x4dea72 (0xfa73 bytes) 000206[0] 1772678745.972886 mapread 0x452000 thru 0x45f833 (0xd834 bytes) 000207[1] 1772678746.000586 mapwrite 0x3a3000 thru 0x3ae291 (0xb292 bytes) 000211[1] 1772678746.146681 mapwrite 0x61d000 thru 0x62354c (0x654d bytes) 000212[0] 1772678746.252631 write_OD 0x069000 thru 0x077fff (0xf000 bytes) 000213[0] 1772678746.344443 mapwrite 0x559000 thru 0x55f6e3 (0x66e4 bytes) 000214[1] 1772678746.402923 mapwrite 0x19c000 thru 0x1a3e76 (0x7e77 bytes) 000216[0] 1772678746.493955 write_OD 0x910000 thru 0x91afff (0xb000 bytes) 000217[1] 1772678746.556100 trunc from 0x91b000 to 0x277188 (0x6a3e77 bytes) 000218[1] 1772678746.597085 write 0x095000 thru 0x09dc27 (0x8c28 bytes) 000219[1] 1772678746.625773 write 0x89a000 thru 0x8a99bf (0xf9c0 bytes) 000220[0] 1772678746.636792 mapread 0x10e000 thru 0x10e8cf (0x8d0 bytes) 000221[0] 1772678746.678511 mapread 0x7d4000 thru 0x7d9649 (0x564a bytes) 000222[0] 1772678746.697475 write_OD 0x5f1000 thru 0x5f3fff (0x3000 bytes) 000224[0] 1772678746.747392 write_OD 0x8f7000 thru 0x8f9fff (0x3000 bytes) 000225[1] 1772678746.777713 read 0x4da000 thru 0x4da06e (0x6f bytes) 000226[0] 1772678746.785568 mapread 0x4f1000 thru 0x4fba4e (0xaa4f bytes) 000227[0] 1772678746.800673 read_OD 0x5c6000 thru 0x5cefff (0x9000 bytes) 000228[0] 1772678746.806700 mapread 0x104000 thru 0x11363a (0xf63b bytes) 000229[1] 1772678746.829478 write 0x8f3000 thru 0x8fdde2 (0xade3 bytes) 000230[1] 1772678746.854061 write 0x30d000 thru 0x31a9a0 (0xd9a1 bytes) 000231[0] 1772678746.879891 mapwrite 0x763000 thru 0x772a28 (0xfa29 bytes) 000233[1] 1772678747.033680 write 0x62f000 thru 0x63ecd8 (0xfcd9 bytes) 000234[1] 1772678747.052843 write 0x6dc000 thru 0x6e37ed (0x77ee bytes) 000235[0] 1772678747.087191 mapwrite 0x51b000 thru 0x52a4f4 (0xf4f5 bytes) 000236[1] 1772678747.209825 mapwrite 0x990000 thru 0x9927f6 (0x27f7 bytes) 000237[1] 1772678747.359661 trunc from 0x9927f7 to 0x27bd96 (0x716a60 bytes) 000238[0] 1772678747.388700 mapwrite 0x73e000 thru 0x74c9e2 (0xe9e3 bytes) 000239[0] 1772678747.476460 trunc from 0x74c9e3 to 0x7a2184 (0x557a2 bytes) 000240[0] 1772678747.495912 read_OD 0x540000 thru 0x548fff (0x9000 bytes) 000241[0] 1772678747.509721 mapread 0x0f6000 thru 0x1031c1 (0xd1c2 bytes) 000242[0] 1772678747.538663 write_OD 0x172000 thru 0x179fff (0x8000 bytes) 000243[1] 1772678747.595806 write 0x6ea000 thru 0x6f3e4e (0x9e4f bytes) 000247[1] 1772678747.648186 mapread 0x00c000 thru 0x00c012 (0x13 bytes) truncating to largest ever: 0x96d381 000250[0] 1772678747.676313 trunc from 0x7a2185 to 0x96d380 (0x1cb1fc bytes) 000252[1] 1772678747.719805 mapwrite 0x51a000 thru 0x52443c (0xa43d bytes) 000253[1] 1772678747.790625 write 0x1ff000 thru 0x20c913 (0xd914 bytes) 000254[1] 1772678747.802730 mapwrite 0x875000 thru 0x87596d (0x96e bytes) 000255[0] 1772678747.843996 mapwrite 0x369000 thru 0x3765fa (0xd5fb bytes) 000258[1] 1772678747.948836 read 0x1d3000 thru 0x1defed (0xbfee bytes) 000259[1] 1772678747.972495 mapread 0x585000 thru 0x5905af (0xb5b0 bytes) 000260[1] 1772678748.014351 read 0x8c7000 thru 0x8d4429 (0xd42a bytes) 000262[1] 1772678748.028023 write 0x795000 thru 0x797523 (0x2524 bytes) 000265[1] 1772678748.077513 mapread 0x292000 thru 0x2a1284 (0xf285 bytes) 000267[1] 1772678748.112342 mapread 0x4a2000 thru 0x4b0d15 (0xed16 bytes) 000268[0] 1772678748.143906 mapwrite 0x7f2000 thru 0x7f3af3 (0x1af4 bytes) 000270[0] 1772678748.264206 mapread 0x661000 thru 0x661a8b (0xa8c bytes) 000271[0] 1772678748.285664 mapread 0x8cb000 thru 0x8da99d (0xf99e bytes) 000272[0] 1772678748.310513 trunc from 0x96d381 to 0x264e7e (0x708502 bytes) 000273[1] 1772678748.335283 read 0x015000 thru 0x0191fc (0x41fd bytes) 000274[0] 1772678748.357581 read_OD 0x094000 thru 0x09efff (0xb000 bytes) 000275[0] 1772678748.369155 write_OD 0x153000 thru 0x15afff (0x8000 bytes) 000276[0] 1772678748.431922 write_OD 0x27d000 thru 0x27efff (0x2000 bytes) 000277[1] 1772678748.476728 mapread 0x116000 thru 0x118ac4 (0x2ac5 bytes) 000278[1] 1772678748.503071 read 0x159000 thru 0x1607ca (0x77cb bytes) 000281[0] 1772678748.533663 mapread 0x0cc000 thru 0x0d134d (0x534e bytes) 000286[1] 1772678748.566079 mapwrite 0x26d000 thru 0x27479b (0x779c bytes) 000287[0] 1772678748.686490 mapwrite 0x0e3000 thru 0x0e46d4 (0x16d5 bytes) 000288[0] 1772678748.762174 mapread 0x1f0000 thru 0x1f23f3 (0x23f4 bytes) 000292[0] 1772678748.812791 trunc from 0x27f000 to 0x53e0e7 (0x2bf0e8 bytes) 000294[1] 1772678748.855067 trunc from 0x53e0e8 to 0x8f755b (0x3b9474 bytes) 000295[1] 1772678748.878953 mapread 0x79f000 thru 0x7a231b (0x331c bytes) 000297[0] 1772678748.973991 write_OD 0x00e000 thru 0x012fff (0x5000 bytes) 000298[1] 1772678749.020965 mapread 0x600000 thru 0x6093ba (0x93bb bytes) 000299[1] 1772678749.054103 read 0x872000 thru 0x881c65 (0xfc66 bytes) 000302[0] 1772678749.063517 trunc from 0x8f755c to 0x3cfc6e (0x5278ed bytes) 000303[1] 1772678749.109676 mapread 0x132000 thru 0x13af24 (0x8f25 bytes) 000304[1] 1772678749.164776 write 0x40a000 thru 0x40ec7e (0x4c7f bytes) 000305[0] 1772678749.177890 write_OD 0x2e5000 thru 0x2e6fff (0x2000 bytes) 000306[0] 1772678749.217796 read_OD 0x3ec000 thru 0x3effff (0x4000 bytes) 000308[0] 1772678749.227227 read_OD 0x312000 thru 0x31ffff (0xe000 bytes) 000309[1] 1772678749.245534 mapwrite 0x793000 thru 0x79ccd5 (0x9cd6 bytes) 000310[1] 1772678749.365641 mapread 0x098000 thru 0x09f01a (0x701b bytes) 000311[0] 1772678749.406104 mapread 0x5a4000 thru 0x5a9002 (0x5003 bytes) 000312[0] 1772678749.462869 trunc from 0x79ccd6 to 0x15686a (0x64646b bytes) 000318[1] 1772678749.526218 write 0x77e000 thru 0x782303 (0x4304 bytes) 000319[0] 1772678749.554951 mapread 0x129000 thru 0x138e3d (0xfe3e bytes) 000320[0] 1772678749.595796 trunc from 0x782304 to 0x51c18c (0x266177 bytes) 000322[0] 1772678749.647749 mapwrite 0x2c6000 thru 0x2d3684 (0xd685 bytes) 000323[1] 1772678749.758948 mapwrite 0x3e8000 thru 0x3f0d93 (0x8d94 bytes) 000324[0] 1772678749.847655 mapwrite 0x124000 thru 0x12a991 (0x6992 bytes) 000328[1] 1772678749.966306 mapread 0x2e0000 thru 0x2e8ee2 (0x8ee3 bytes) 000333[1] 1772678749.998434 mapread 0x07a000 thru 0x086847 (0xc848 bytes) 000336[1] 1772678750.049605 write 0x133000 thru 0x1355d0 (0x25d1 bytes) 000337[1] 1772678750.072976 read 0x356000 thru 0x35d088 (0x7089 bytes) 000338[1] 1772678750.080324 trunc from 0x51c18d to 0x1e9a32 (0x33275a bytes) 000342[1] 1772678750.113232 read 0x091000 thru 0x0a0867 (0xf868 bytes) 000343[0] 1772678750.144696 mapread 0x0d2000 thru 0x0dcc5a (0xac5b bytes) 000344[0] 1772678750.207871 trunc from 0x1e9a33 to 0x7602c9 (0x576897 bytes) 000345[1] 1772678750.249977 write 0x2c0000 thru 0x2c7dca (0x7dcb bytes) 000346[0] 1772678750.297296 write_OD 0x93e000 thru 0x940fff (0x3000 bytes) 000347[1] 1772678750.371862 read 0x8c5000 thru 0x8cd14b (0x814c bytes) 000348[1] 1772678750.389268 mapread 0x23a000 thru 0x23c0ae (0x20af bytes) 000354[0] 1772678750.420325 write_OD 0x0ef000 thru 0x0f9fff (0xb000 bytes) 000357[1] 1772678750.504110 mapread 0x2bb000 thru 0x2c138b (0x638c bytes) 000361[0] 1772678750.520595 read_OD 0x2d2000 thru 0x2d3fff (0x2000 bytes) 000363[0] 1772678750.528682 mapwrite 0x970000 thru 0x97256e (0x256f bytes) 000365[0] 1772678750.630638 write_OD 0x6ad000 thru 0x6b1fff (0x5000 bytes) 000369[0] 1772678750.670122 write_OD 0x761000 thru 0x76dfff (0xd000 bytes) 000373[1] 1772678750.718531 write 0x3b2000 thru 0x3c0aed (0xeaee bytes) 000374[1] 1772678750.756625 mapread 0x710000 thru 0x714607 (0x4608 bytes) 000375[0] 1772678750.791501 mapread 0x0da000 thru 0x0e4614 (0xa615 bytes) 000378[0] 1772678750.846318 mapread 0x4f0000 thru 0x4fcdca (0xcdcb bytes) 000379[1] 1772678750.896281 read 0x582000 thru 0x58bd4d (0x9d4e bytes) 000380[1] 1772678750.903990 mapread 0x022000 thru 0x029e71 (0x7e72 bytes) 000385[1] 1772678750.929650 mapwrite 0x9f6000 thru 0x9f8d07 (0x2d08 bytes) 000386[0] 1772678751.020386 trunc from 0x9f8d08 to 0x508c13 (0x4f00f4 bytes) 000387[0] 1772678751.035540 trunc from 0x508c14 to 0x6c5721 (0x1bcb0e bytes) 000390[0] 1772678751.090709 read_OD 0x476000 thru 0x478fff (0x3000 bytes) 000392[1] 1772678751.109877 read 0x542000 thru 0x54f6d0 (0xd6d1 bytes) 000394[0] 1772678751.117297 read_OD 0x540000 thru 0x542fff (0x3000 bytes) 000398[0] 1772678751.123587 trunc from 0x6c5722 to 0x0f37ab (0x5d1f76 bytes) 000399[1] 1772678751.175513 trunc from 0x0f37ac to 0x95faec (0x86c341 bytes) 000402[1] 1772678751.229234 mapwrite 0x3ba000 thru 0x3c5e91 (0xbe92 bytes) 000403[0] 1772678751.292448 mapread 0x046000 thru 0x0525ce (0xc5cf bytes) 000406[1] 1772678751.321654 trunc from 0x95faed to 0x624299 (0x33b853 bytes) 000409[1] 1772678751.381325 mapwrite 0x261000 thru 0x27050d (0xf50e bytes) 000410[1] 1772678751.428624 read 0x167000 thru 0x16cd33 (0x5d34 bytes) 000412[1] 1772678751.457608 mapwrite 0x044000 thru 0x0526f4 (0xe6f5 bytes) 000415[0] 1772678751.502347 read_OD 0x265000 thru 0x26cfff (0x8000 bytes) 000418[0] 1772678751.514665 write_OD 0x5d2000 thru 0x5d8fff (0x7000 bytes) 000419[0] 1772678751.536751 write_OD 0x837000 thru 0x837fff (0x1000 bytes) 000422[0] 1772678751.563701 write_OD 0x940000 thru 0x948fff (0x9000 bytes) 000423[1] 1772678751.594797 trunc from 0x949000 to 0x10a4b2 (0x83eb4d bytes) 000424[1] 1772678751.612777 write 0x06d000 thru 0x074139 (0x713a bytes) 000426[1] 1772678751.627800 mapread 0x02e000 thru 0x035728 (0x7729 bytes) 000428[1] 1772678751.647484 read 0x0cc000 thru 0x0d6fd6 (0xafd7 bytes) 000431[0] 1772678751.658073 mapread 0x030000 thru 0x032257 (0x2258 bytes) 000432[1] 1772678751.683003 trunc from 0x10a4b3 to 0x6e9d74 (0x5df8c2 bytes) 000433[0] 1772678751.720781 mapwrite 0x9f9000 thru 0x9ff28a (0x628b bytes) 000435[0] 1772678751.816868 mapread 0x590000 thru 0x5978dd (0x78de bytes) 000436[1] 1772678751.846129 mapread 0x881000 thru 0x888743 (0x7744 bytes) 000437[0] 1772678751.918879 mapwrite 0x2aa000 thru 0x2adab0 (0x3ab1 bytes) 000439[1] 1772678751.987711 mapread 0x5e5000 thru 0x5f4dfb (0xfdfc bytes) 000440[0] 1772678752.063778 mapwrite 0x67c000 thru 0x689ae2 (0xdae3 bytes) 000441[1] 1772678752.130922 read 0x40f000 thru 0x41aadd (0xbade bytes) 000443[1] 1772678752.147997 mapwrite 0x63a000 thru 0x644638 (0xa639 bytes) 000444[0] 1772678752.205796 write_OD 0x12d000 thru 0x130fff (0x4000 bytes) 000445[0] 1772678752.239241 read_OD 0x703000 thru 0x70cfff (0xa000 bytes) 000446[0] 1772678752.246711 mapwrite 0x87b000 thru 0x8857e3 (0xa7e4 bytes) 000447[1] 1772678752.309811 read 0x796000 thru 0x79ad41 (0x4d42 bytes) 000448[0] 1772678752.322163 trunc from 0x9ff28b to 0x3b2a27 (0x64c863 bytes) 000452[0] 1772678752.347776 mapwrite 0x043000 thru 0x050ad1 (0xdad2 bytes) 000453[1] 1772678752.395253 read 0x315000 thru 0x324770 (0xf771 bytes) 000454[0] 1772678752.422895 write_OD 0x30f000 thru 0x316fff (0x8000 bytes) 000455[1] 1772678752.457828 mapread 0x034000 thru 0x040b93 (0xcb94 bytes) 000456[0] 1772678752.497369 write_OD 0x711000 thru 0x71cfff (0xc000 bytes) 000457[0] 1772678752.527673 trunc from 0x71d000 to 0x437795 (0x2e586a bytes) 000458[0] 1772678752.547680 trunc from 0x437796 to 0x71e644 (0x2e6eaf bytes) 000460[1] 1772678752.578505 mapwrite 0x1d4000 thru 0x1deeed (0xaeee bytes) 000465[0] 1772678752.680643 read_OD 0x47f000 thru 0x486fff (0x8000 bytes) 000466[1] 1772678752.693553 mapwrite 0x648000 thru 0x654c88 (0xcc89 bytes) 000467[0] 1772678752.743907 trunc from 0x71e645 to 0x34397d (0x3dacc7 bytes) 000468[1] 1772678752.761895 mapwrite 0x546000 thru 0x54bf9b (0x5f9c bytes) 000470[1] 1772678752.818476 mapwrite 0x656000 thru 0x6568f1 (0x8f2 bytes) 000471[1] 1772678752.848658 mapread 0x4dc000 thru 0x4e327a (0x727b bytes) 000473[0] 1772678752.886772 read_OD 0x42e000 thru 0x434fff (0x7000 bytes) 000475[1] 1772678752.898406 mapread 0x01b000 thru 0x027330 (0xc331 bytes) 000477[0] 1772678752.920482 trunc from 0x6568f2 to 0x36b667 (0x2eb28a bytes) 000479[1] 1772678752.942410 mapread 0x15b000 thru 0x169b3a (0xeb3b bytes) 000480[0] 1772678752.978858 read_OD 0x32a000 thru 0x32bfff (0x2000 bytes) 000484[0] 1772678752.984835 mapwrite 0x630000 thru 0x631e03 (0x1e04 bytes) 000486[0] 1772678753.073914 mapwrite 0x96e000 thru 0x97cb7b (0xeb7c bytes) 000487[0] 1772678753.133675 trunc from 0x97cb7c to 0x81c29e (0x1608dd bytes) 000488[0] 1772678753.140753 write_OD 0x2ff000 thru 0x307fff (0x9000 bytes) 000490[1] 1772678753.175471 read 0x64f000 thru 0x65ee7d (0xfe7e bytes) 000491[0] 1772678753.212696 read_OD 0x286000 thru 0x289fff (0x4000 bytes) 000492[1] 1772678753.215381 trunc from 0x81c29f to 0x783403 (0x98e9b bytes) 000495[0] 1772678753.256783 trunc from 0x783404 to 0x95ef1e (0x1dbb1b bytes) 000499[0] 1772678753.291695 mapwrite 0x270000 thru 0x278115 (0x8116 bytes) 000500[1] 1772678753.338004 mapwrite 0x611000 thru 0x61f8f4 (0xe8f5 bytes) All operations completed A-OK! PASS 16a (32s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 21:45:55 (1772678755) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.245605 s, 34.2 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.148251 s, 56.6 MB/s Chance of close/open is 1 in 50 Seed set to 8757 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000004[1] 1772678757.739298 write 0x90456d thru 0x904c90 (0x724 bytes) 000005[0] 1772678757.766960 mapread 0x8530f1 thru 0x853f70 (0xe80 bytes) truncating to largest ever: 0x479c9d 000006[1] 1772678757.786645 trunc from 0x904c91 to 0x479c9c (0x48aff4 bytes) 000007[0] 1772678757.821986 read 0x12aff1 thru 0x12b318 (0x328 bytes) 000008[0] 1772678757.831632 mapwrite 0x6f7964 thru 0x6f8bda (0x1277 bytes) 000009[0] 1772678757.871099 mapread 0x2ada65 thru 0x2ae03e (0x5da bytes) 000013[0] 1772678757.889550 read 0x030370 thru 0x032289 (0x1f1a bytes) 000015[1] 1772678757.898319 mapwrite 0x56599d thru 0x566181 (0x7e5 bytes) 000016[1] 1772678757.938476 read 0x658133 thru 0x658cc1 (0xb8f bytes) 000017[1] 1772678757.946883 mapread 0x39db13 thru 0x39de65 (0x353 bytes) truncating to largest ever: 0x576eda 000018[1] 1772678757.960342 trunc from 0x6f8bdb to 0x576ed9 (0x181d01 bytes) 000019[0] 1772678757.978867 mapwrite 0x87eac7 thru 0x88029c (0x17d6 bytes) 000020[1] 1772678758.050474 read 0x5abd12 thru 0x5ac18b (0x47a bytes) 000022[1] 1772678758.074779 read 0x0a0dcf thru 0x0a0fa9 (0x1db bytes) 000023[1] 1772678758.085191 mapread 0x0cbc06 thru 0x0ccd84 (0x117f bytes) 000024[1] 1772678758.089912 trunc from 0x88029d to 0x1f352d (0x68cd6f bytes) 000026[0] 1772678758.120132 trunc from 0x1f352e to 0x198dcf (0x5a75e bytes) truncating to largest ever: 0x89dd62 000030[1] 1772678758.181754 trunc from 0x198dd0 to 0x89dd61 (0x704f92 bytes) 000031[1] 1772678758.202387 mapread 0x738d57 thru 0x738de2 (0x8c bytes) 000032[0] 1772678758.227764 mapread 0x77d0eb thru 0x77d727 (0x63d bytes) 000033[0] 1772678758.246911 write 0x5a5c33 thru 0x5a7408 (0x17d6 bytes) 000034[0] 1772678758.267153 trunc from 0x89dd62 to 0x4fdf67 (0x39fdfa bytes) 000035[0] 1772678758.295172 mapread 0x296aa2 thru 0x2978c6 (0xe25 bytes) 000036[1] 1772678758.304860 mapread 0x0d24c9 thru 0x0d36df (0x1217 bytes) 000037[1] 1772678758.328866 read 0x205bbf thru 0x207119 (0x155b bytes) 000038[0] 1772678758.335338 read 0x1062e0 thru 0x10762d (0x134e bytes) 000039[0] 1772678758.355490 mapwrite 0x7b7449 thru 0x7b806f (0xc27 bytes) 000040[1] 1772678758.445016 write 0x8337cd thru 0x835044 (0x1878 bytes) 000041[0] 1772678758.470506 write 0x4a39c6 thru 0x4a457f (0xbba bytes) 000042[1] 1772678758.487694 write 0x3d55db thru 0x3d7377 (0x1d9d bytes) 000043[0] 1772678758.507658 read 0x391740 thru 0x39353c (0x1dfd bytes) 000045[1] 1772678758.524852 read 0x24a0d6 thru 0x24bf3d (0x1e68 bytes) 000047[0] 1772678758.555779 read 0x259d85 thru 0x25ad0f (0xf8b bytes) 000048[0] 1772678758.571527 trunc from 0x835045 to 0x2e5b0d (0x54f537 bytes) 000049[0] 1772678758.590836 trunc from 0x2e5b0e to 0x2a5428 (0x406e5 bytes) 000053[1] 1772678758.643550 write 0x29fe23 thru 0x2a1cba (0x1e98 bytes) 000054[1] 1772678758.661238 read 0x23aada thru 0x23b9a3 (0xeca bytes) 000056[1] 1772678758.723151 mapwrite 0x87352e thru 0x8751f2 (0x1cc5 bytes) 000057[0] 1772678758.817888 write 0x31e099 thru 0x31ee06 (0xd6e bytes) 000058[1] 1772678758.838165 mapread 0x23c87a thru 0x23d541 (0xcc8 bytes) 000059[1] 1772678758.861646 write 0x7ef3bb thru 0x7efd45 (0x98b bytes) 000061[1] 1772678758.863665 write 0x453f7b thru 0x455013 (0x1099 bytes) 000062[0] 1772678758.864919 write 0x0fc25e thru 0x0fd4e3 (0x1286 bytes) 000064[1] 1772678758.896557 write 0x3353bd thru 0x3358dc (0x520 bytes) 000068[0] 1772678758.921982 mapwrite 0x3b9b6b thru 0x3baf5d (0x13f3 bytes) 000069[1] 1772678758.971772 mapread 0x05c3c4 thru 0x05d99d (0x15da bytes) 000070[1] 1772678759.016875 mapwrite 0x7099ae thru 0x70b0e4 (0x1737 bytes) 000072[1] 1772678759.068471 write 0x28bcfa thru 0x28dbde (0x1ee5 bytes) 000073[1] 1772678759.084825 mapread 0x2ef841 thru 0x2efb29 (0x2e9 bytes) 000074[0] 1772678759.096681 read 0x052249 thru 0x0525df (0x397 bytes) 000080[0] 1772678759.127577 write 0x2e270b thru 0x2e31bc (0xab2 bytes) 000081[0] 1772678759.142620 mapread 0x5ab5e8 thru 0x5ac451 (0xe6a bytes) 000082[1] 1772678759.173343 write 0x40af57 thru 0x40c33b (0x13e5 bytes) 000083[0] 1772678759.200216 read 0x6f96b0 thru 0x6fb0af (0x1a00 bytes) 000086[1] 1772678759.251594 mapread 0x62434f thru 0x62627c (0x1f2e bytes) 000087[1] 1772678759.276264 read 0x13d0e8 thru 0x13e901 (0x181a bytes) 000088[0] 1772678759.288589 write 0x703c9b thru 0x70412e (0x494 bytes) 000090[0] 1772678759.327161 write 0x87cd04 thru 0x87cea8 (0x1a5 bytes) 000091[0] 1772678759.334579 mapwrite 0x50095c thru 0x5019ad (0x1052 bytes) 000094[0] 1772678759.358830 mapread 0x286251 thru 0x287f5b (0x1d0b bytes) 000100[0] 1772678759.441821 trunc from 0x87cea9 to 0x128108 (0x754da0 bytes) 000102[0] 1772678759.458566 mapread 0x074a3f thru 0x075c8c (0x124e bytes) 000104[0] 1772678759.468515 mapread 0x04a409 thru 0x04baaf (0x16a7 bytes) 000105[1] 1772678759.477381 mapread 0x10aa9e thru 0x10b733 (0xc96 bytes) 000108[1] 1772678759.534421 read 0x101237 thru 0x101943 (0x70d bytes) 000110[0] 1772678759.570367 write 0x2c70fe thru 0x2c8613 (0x1516 bytes) 000111[0] 1772678759.598150 write 0x88a76f thru 0x88b15d (0x9ef bytes) 000112[0] 1772678759.621764 mapwrite 0x2523b5 thru 0x253ca1 (0x18ed bytes) 000113[0] 1772678759.690619 mapread 0x19b85b thru 0x19c58a (0xd30 bytes) 000120[0] 1772678759.755001 mapwrite 0x705123 thru 0x7070db (0x1fb9 bytes) 000122[0] 1772678759.808190 mapwrite 0x547e22 thru 0x549919 (0x1af8 bytes) 000123[0] 1772678759.845054 mapread 0x39aae4 thru 0x39b5d9 (0xaf6 bytes) 000124[0] 1772678759.851147 mapread 0x79c7fe thru 0x79d6a7 (0xeaa bytes) 000127[0] 1772678759.869443 write 0x131e0b thru 0x131f6b (0x161 bytes) 000128[0] 1772678759.881806 read 0x70efba thru 0x71026d (0x12b4 bytes) 000129[1] 1772678759.898778 read 0x6fca95 thru 0x6fd8ac (0xe18 bytes) 000131[0] 1772678759.934006 read 0x7a2570 thru 0x7a27bd (0x24e bytes) 000132[0] 1772678759.950615 mapwrite 0x0e2498 thru 0x0e3b7e (0x16e7 bytes) 000133[1] 1772678759.982436 mapread 0x5c9bf8 thru 0x5cba4a (0x1e53 bytes) 000135[0] 1772678760.014791 mapwrite 0x4c73d0 thru 0x4c797d (0x5ae bytes) 000139[1] 1772678760.116811 read 0x41fd00 thru 0x42144f (0x1750 bytes) 000140[1] 1772678760.142862 write 0x34c195 thru 0x34e073 (0x1edf bytes) 000143[1] 1772678760.212582 read 0x32b098 thru 0x32bdfa (0xd63 bytes) truncating to largest ever: 0x9830c9 000146[1] 1772678760.224808 trunc from 0x88b15e to 0x9830c8 (0xf7f6b bytes) 000151[0] 1772678760.343610 write 0x04557f thru 0x04619a (0xc1c bytes) 000152[0] 1772678760.380261 trunc from 0x9830c9 to 0x5f9593 (0x389b35 bytes) 000153[1] 1772678760.397233 trunc from 0x5f9594 to 0x826ca9 (0x22d716 bytes) 000154[1] 1772678760.425133 mapwrite 0x239c48 thru 0x23afd9 (0x1392 bytes) 000155[0] 1772678760.494201 trunc from 0x826caa to 0x6fb6fc (0x12b5ad bytes) 000158[1] 1772678760.551152 write 0x5f5564 thru 0x5f58e4 (0x381 bytes) 000160[1] 1772678760.582118 mapread 0x15579f thru 0x15692d (0x118f bytes) 000161[0] 1772678760.631668 trunc from 0x6fb6fd to 0x440bad (0x2bab4f bytes) 000162[1] 1772678760.679297 mapwrite 0x21fdce thru 0x22187c (0x1aaf bytes) 000163[0] 1772678760.738554 write 0x04da2f thru 0x04f3c9 (0x199b bytes) 000165[0] 1772678760.766010 trunc from 0x440bae to 0x88df01 (0x44d354 bytes) 000166[0] 1772678760.795250 trunc from 0x88df02 to 0x66ced4 (0x22102d bytes) 000167[0] 1772678760.806422 trunc from 0x66ced5 to 0x38a7fe (0x2e26d6 bytes) 000170[1] 1772678760.841793 mapread 0x185ddd thru 0x185dfc (0x20 bytes) 000173[0] 1772678760.906378 write 0x4900e0 thru 0x4905ca (0x4eb bytes) 000176[1] 1772678760.920478 trunc from 0x4905cb to 0x3f1b4f (0x9ea7b bytes) 000177[1] 1772678760.949273 trunc from 0x3f1b50 to 0x2c3be4 (0x12df6b bytes) 000179[0] 1772678760.960453 trunc from 0x2c3be5 to 0x3799b8 (0xb5dd4 bytes) 000180[0] 1772678760.989520 read 0x1e63f0 thru 0x1e76d7 (0x12e8 bytes) 000182[1] 1772678761.053831 mapwrite 0x336071 thru 0x3363e5 (0x375 bytes) 000183[1] 1772678761.130678 trunc from 0x3799b9 to 0x802431 (0x488a79 bytes) 000185[0] 1772678761.185407 trunc from 0x802432 to 0x72d481 (0xd4fb0 bytes) 000188[0] 1772678761.221762 trunc from 0x72d482 to 0x4f5836 (0x237c4b bytes) 000189[1] 1772678761.232613 mapread 0x3cb2ca thru 0x3cc3ce (0x1105 bytes) 000191[0] 1772678761.278346 read 0x0c5739 thru 0x0c7317 (0x1bdf bytes) 000192[0] 1772678761.283985 read 0x1cc735 thru 0x1ccf36 (0x802 bytes) 000193[1] 1772678761.288379 mapread 0x0f6ac2 thru 0x0f7771 (0xcb0 bytes) 000196[1] 1772678761.312322 write 0x7010f7 thru 0x701666 (0x570 bytes) 000201[1] 1772678761.339288 mapwrite 0x7305cc thru 0x7323b2 (0x1de7 bytes) 000202[1] 1772678761.405814 read 0x54de7b thru 0x54f13e (0x12c4 bytes) 000203[0] 1772678761.423888 read 0x1c9202 thru 0x1ca30b (0x110a bytes) 000204[1] 1772678761.444816 write 0x467286 thru 0x468e69 (0x1be4 bytes) 000208[1] 1772678761.479365 read 0x06590d thru 0x066f01 (0x15f5 bytes) 000209[1] 1772678761.485774 trunc from 0x7323b3 to 0x96362c (0x23127a bytes) 000210[1] 1772678761.511501 read 0x4a66b9 thru 0x4a7958 (0x12a0 bytes) 000211[1] 1772678761.516477 trunc from 0x96362d to 0x757c46 (0x20b9e6 bytes) 000213[1] 1772678761.534369 write 0x299f96 thru 0x29bf83 (0x1fee bytes) 000215[1] 1772678761.569154 write 0x742839 thru 0x743bd7 (0x139f bytes) 000216[0] 1772678761.576624 mapwrite 0x12ece7 thru 0x13055c (0x1876 bytes) 000217[1] 1772678761.664943 read 0x36d5d6 thru 0x36e704 (0x112f bytes) 000218[1] 1772678761.699911 trunc from 0x757c47 to 0x0a0a91 (0x6b71b5 bytes) 000220[1] 1772678761.754506 read 0x070308 thru 0x071ab5 (0x17ae bytes) 000221[0] 1772678761.770891 mapwrite 0x8fc073 thru 0x8fc616 (0x5a4 bytes) 000222[0] 1772678761.842962 trunc from 0x8fc617 to 0x95fed4 (0x638be bytes) 000229[0] 1772678761.893989 mapwrite 0x1fbedc thru 0x1fd032 (0x1157 bytes) 000234[0] 1772678761.978515 read 0x3c611e thru 0x3c76b3 (0x1596 bytes) 000235[0] 1772678761.992528 mapwrite 0x743c32 thru 0x745237 (0x1606 bytes) 000239[0] 1772678762.047655 mapwrite 0x11b23f thru 0x11bcc2 (0xa84 bytes) 000242[0] 1772678762.107494 trunc from 0x95fed5 to 0x0c7f63 (0x897f71 bytes) 000243[0] 1772678762.134340 write 0x1a8546 thru 0x1a8bae (0x669 bytes) 000244[1] 1772678762.135687 mapread 0x0cde67 thru 0x0cef27 (0x10c1 bytes) 000249[1] 1772678762.219718 mapwrite 0x7f9c00 thru 0x7fb6a7 (0x1aa8 bytes) 000251[0] 1772678762.418300 trunc from 0x7fb6a8 to 0x799dc5 (0x618e2 bytes) 000252[0] 1772678762.459509 mapwrite 0x6d5c6f thru 0x6d704b (0x13dd bytes) 000254[1] 1772678762.536609 trunc from 0x799dc6 to 0x681c0f (0x1181b6 bytes) 000258[0] 1772678762.664993 mapwrite 0x6aa3c1 thru 0x6abdde (0x1a1e bytes) 000259[1] 1772678762.792162 mapread 0x629847 thru 0x62a19b (0x955 bytes) 000263[0] 1772678762.852357 mapwrite 0x68792f thru 0x6887be (0xe90 bytes) 000266[0] 1772678762.940722 mapwrite 0x066a06 thru 0x0684f7 (0x1af2 bytes) 000269[0] 1772678763.024288 write 0x265cd2 thru 0x265fbe (0x2ed bytes) 000272[0] 1772678763.034140 trunc from 0x6abddf to 0x29bce9 (0x4100f5 bytes) 000275[1] 1772678763.068592 write 0x9fd125 thru 0x9fd28f (0x16b bytes) 000276[1] 1772678763.119821 mapwrite 0x2f08dc thru 0x2f16eb (0xe10 bytes) 000280[1] 1772678763.300356 mapread 0x3a8fef thru 0x3aa626 (0x1638 bytes) 000284[1] 1772678763.417880 mapwrite 0x2d3959 thru 0x2d4f2e (0x15d6 bytes) 000285[1] 1772678763.520341 mapwrite 0x59d8c0 thru 0x59f615 (0x1d56 bytes) 000286[1] 1772678763.591865 mapread 0x0e5c87 thru 0x0e70e1 (0x145b bytes) 000287[1] 1772678763.606397 mapwrite 0x4b7ed7 thru 0x4b8b83 (0xcad bytes) 000290[0] 1772678763.758745 trunc from 0x9fd290 to 0x43481a (0x5c8a75 bytes) 000293[1] 1772678763.810948 mapwrite 0x27cf25 thru 0x27e33b (0x1417 bytes) 000294[1] 1772678763.909901 mapwrite 0x560c6f thru 0x56234e (0x16e0 bytes) 000295[1] 1772678764.047705 trunc from 0x56234f to 0x8a8296 (0x345f48 bytes) 000297[0] 1772678764.084419 read 0x18e68b thru 0x18fd78 (0x16ee bytes) 000301[1] 1772678764.101698 mapread 0x2663a7 thru 0x266e7b (0xad5 bytes) 000302[1] 1772678764.117475 mapread 0x36d05c thru 0x36e2c0 (0x1265 bytes) 000303[1] 1772678764.124929 trunc from 0x8a8297 to 0x7dc164 (0xcc132 bytes) 000304[1] 1772678764.165650 trunc from 0x7dc165 to 0x7873af (0x54db5 bytes) 000307[1] 1772678764.238619 mapread 0x5fbd40 thru 0x5fc864 (0xb25 bytes) 000309[0] 1772678764.272634 trunc from 0x7873b0 to 0x724417 (0x62f98 bytes) 000312[0] 1772678764.360632 write 0x0d2108 thru 0x0d40b3 (0x1fac bytes) 000313[1] 1772678764.370356 read 0x65bb84 thru 0x65d4da (0x1957 bytes) 000314[1] 1772678764.386907 write 0x8a4b38 thru 0x8a6b03 (0x1fcc bytes) 000316[0] 1772678764.428894 mapwrite 0x198aca thru 0x1999fb (0xf32 bytes) 000319[1] 1772678764.523454 mapread 0x64a96a thru 0x64ad68 (0x3ff bytes) 000320[1] 1772678764.554378 mapwrite 0x1d3496 thru 0x1d50da (0x1c45 bytes) 000321[1] 1772678764.628696 trunc from 0x8a6b04 to 0x018ce4 (0x88de1f bytes) 000322[1] 1772678764.670510 trunc from 0x018ce5 to 0x5bdea9 (0x5a51c5 bytes) 000324[1] 1772678764.694078 mapread 0x2130b9 thru 0x214816 (0x175e bytes) 000329[0] 1772678764.776801 read 0x042351 thru 0x043348 (0xff8 bytes) 000330[0] 1772678764.812308 write 0x351ed4 thru 0x352f5e (0x108b bytes) 000332[1] 1772678764.878674 mapwrite 0x67a63e thru 0x67b185 (0xb48 bytes) 000334[0] 1772678765.006787 read 0x2e30e3 thru 0x2e4c7d (0x1b9b bytes) 000335[1] 1772678765.035813 read 0x519359 thru 0x51aec5 (0x1b6d bytes) 000336[0] 1772678765.049792 mapread 0x2c6c74 thru 0x2c7d89 (0x1116 bytes) 000337[0] 1772678765.081565 mapread 0x470c9c thru 0x4714f5 (0x85a bytes) 000340[0] 1772678765.109735 mapwrite 0x6f4f18 thru 0x6f6cfd (0x1de6 bytes) 000342[1] 1772678765.231922 mapwrite 0x12473a thru 0x12625f (0x1b26 bytes) 000343[0] 1772678765.354961 mapwrite 0x067d20 thru 0x06924c (0x152d bytes) 000349[1] 1772678765.550388 mapread 0x248deb thru 0x24969e (0x8b4 bytes) 000354[1] 1772678765.638637 trunc from 0x6f6cfe to 0x737c44 (0x40f47 bytes) 000355[1] 1772678765.681798 trunc from 0x737c45 to 0x4dba40 (0x25c204 bytes) truncating to largest ever: 0x9a53be 000357[0] 1772678765.702939 trunc from 0x4dba41 to 0x9a53bd (0x4c997d bytes) 000361[0] 1772678765.791080 read 0x1f9fd6 thru 0x1fa31b (0x346 bytes) 000364[1] 1772678765.860432 trunc from 0x9a53be to 0x52730f (0x47e0ae bytes) 000365[1] 1772678765.906887 read 0x3d2e5a thru 0x3d4a29 (0x1bd0 bytes) 000367[1] 1772678765.966319 mapwrite 0x0efce3 thru 0x0f0605 (0x923 bytes) 000368[0] 1772678766.024911 mapwrite 0x2680ab thru 0x268c4d (0xba3 bytes) 000369[0] 1772678766.104854 read 0x184eaf thru 0x186165 (0x12b7 bytes) 000371[1] 1772678766.155522 read 0x1b1ff6 thru 0x1b2c16 (0xc21 bytes) 000372[0] 1772678766.189447 mapread 0x4ec15e thru 0x4ecce4 (0xb87 bytes) 000376[1] 1772678766.215397 mapread 0x4e0b58 thru 0x4e1506 (0x9af bytes) 000380[1] 1772678766.228457 mapread 0x170df8 thru 0x172a00 (0x1c09 bytes) 000381[1] 1772678766.242078 mapwrite 0x7a21c4 thru 0x7a24d0 (0x30d bytes) 000382[1] 1772678766.350334 mapwrite 0x171dda thru 0x172f9d (0x11c4 bytes) 000384[1] 1772678766.404741 trunc from 0x7a24d1 to 0x019db4 (0x78871c bytes) 000385[1] 1772678766.430544 mapread 0x0152fe thru 0x0163c7 (0x10ca bytes) 000386[1] 1772678766.441563 mapread 0x009d78 thru 0x00b336 (0x15bf bytes) 000387[1] 1772678766.463027 read 0x0026cb thru 0x00301d (0x953 bytes) 000389[1] 1772678766.485278 read 0x013300 thru 0x014e47 (0x1b48 bytes) 000390[0] 1772678766.499110 read 0x000b05 thru 0x0027b8 (0x1cb4 bytes) truncating to largest ever: 0x9d2d94 000391[1] 1772678766.539476 trunc from 0x019db5 to 0x9d2d93 (0x9b8fdf bytes) 000397[0] 1772678766.634359 trunc from 0x9d2d94 to 0x9a491d (0x2e476 bytes) 000399[0] 1772678766.689130 write 0x670835 thru 0x67097a (0x146 bytes) 000402[1] 1772678766.788769 mapread 0x2b614d thru 0x2b7d9f (0x1c53 bytes) 000404[0] 1772678766.896890 read 0x82d59a thru 0x82e256 (0xcbd bytes) 000412[1] 1772678766.956485 write 0x7f981f thru 0x7fa946 (0x1128 bytes) 000413[0] 1772678767.071793 write 0x212c08 thru 0x213058 (0x451 bytes) 000414[1] 1772678767.115512 write 0x062da2 thru 0x06362f (0x88e bytes) 000417[0] 1772678767.186682 read 0x3edd6b thru 0x3ef4a1 (0x1737 bytes) 000419[0] 1772678767.229779 mapread 0x97b6aa thru 0x97c606 (0xf5d bytes) 000420[1] 1772678767.271752 write 0x1acdc9 thru 0x1acf4b (0x183 bytes) 000423[1] 1772678767.331675 trunc from 0x9a491e to 0x5842e7 (0x420636 bytes) 000426[1] 1772678767.393009 trunc from 0x5842e8 to 0x6bf677 (0x13b390 bytes) 000429[0] 1772678767.440950 mapread 0x072e35 thru 0x07337f (0x54b bytes) 000430[1] 1772678767.486826 mapread 0x38476c thru 0x38579f (0x1034 bytes) 000431[1] 1772678767.519864 trunc from 0x6bf678 to 0x1ac4f0 (0x513187 bytes) 000433[1] 1772678767.560291 read 0x0a4a45 thru 0x0a58e0 (0xe9c bytes) 000436[0] 1772678767.600785 write 0x2f5caf thru 0x2f67fc (0xb4e bytes) 000439[0] 1772678767.651060 write 0x3a71ad thru 0x3a911d (0x1f71 bytes) 000440[0] 1772678767.672582 trunc from 0x3a911e to 0x9a07d1 (0x5f76b4 bytes) 000441[0] 1772678767.727236 mapread 0x111f27 thru 0x113ecf (0x1fa9 bytes) 000442[1] 1772678767.752650 trunc from 0x9a07d2 to 0x3d1fee (0x5ce7e3 bytes) 000445[1] 1772678767.785360 write 0x9c42ca thru 0x9c61f3 (0x1f2a bytes) 000447[1] 1772678767.803772 write 0x90b498 thru 0x90c3aa (0xf13 bytes) 000453[1] 1772678767.928628 read 0x50a31e thru 0x50b979 (0x165c bytes) 000454[1] 1772678767.929070 trunc from 0x9c61f4 to 0x7f1547 (0x1d4cac bytes) 000455[0] 1772678767.969430 write 0x27bf5e thru 0x27cc55 (0xcf8 bytes) 000457[1] 1772678768.016525 trunc from 0x7f1548 to 0x3864c0 (0x46b087 bytes) 000458[1] 1772678768.062417 write 0x90d69b thru 0x90e115 (0xa7b bytes) 000463[1] 1772678768.083747 mapwrite 0x9a17c1 thru 0x9a1b67 (0x3a7 bytes) 000465[1] 1772678768.153815 read 0x33378e thru 0x334de9 (0x165c bytes) 000466[0] 1772678768.159503 write 0x47b7c6 thru 0x47d74a (0x1f85 bytes) 000467[1] 1772678768.196886 trunc from 0x9a1b68 to 0x8e108c (0xc0adb bytes) 000468[1] 1772678768.257333 mapwrite 0x01d345 thru 0x01e608 (0x12c4 bytes) 000470[1] 1772678768.318223 mapwrite 0x98e3e4 thru 0x98fc15 (0x1832 bytes) 000471[1] 1772678768.435786 read 0x2359e7 thru 0x235add (0xf7 bytes) 000472[0] 1772678768.465816 trunc from 0x98fc16 to 0x335c28 (0x659fed bytes) 000473[0] 1772678768.547774 write 0x5c74d5 thru 0x5c814e (0xc7a bytes) 000477[0] 1772678768.608631 mapwrite 0x64d220 thru 0x64d8fd (0x6de bytes) 000478[1] 1772678768.669723 mapwrite 0x9fa316 thru 0x9fa88f (0x57a bytes) 000480[1] 1772678768.755907 read 0x92be40 thru 0x92d0b0 (0x1271 bytes) 000481[1] 1772678768.767211 mapread 0x06ac1e thru 0x06bef3 (0x12d6 bytes) 000483[0] 1772678768.827271 mapread 0x1e26cf thru 0x1e3f89 (0x18bb bytes) 000484[1] 1772678768.871826 trunc from 0x9fa890 to 0x24b080 (0x7af80f bytes) 000485[0] 1772678768.915727 read 0x098dae thru 0x099f7a (0x11cd bytes) 000486[0] 1772678768.979949 mapread 0x03cd6f thru 0x03d4fc (0x78e bytes) 000487[1] 1772678768.994392 mapread 0x15f9cc thru 0x160f51 (0x1586 bytes) 000488[0] 1772678769.022947 mapwrite 0x6f7186 thru 0x6f842b (0x12a6 bytes) 000489[1] 1772678769.154462 write 0x50731a thru 0x5075c5 (0x2ac bytes) 000490[1] 1772678769.178334 write 0x221033 thru 0x222a5e (0x1a2c bytes) 000496[0] 1772678769.249651 read 0x54b860 thru 0x54cd7d (0x151e bytes) 000498[0] 1772678769.285272 read 0x217aa0 thru 0x219a84 (0x1fe5 bytes) 000500[0] 1772678769.307714 write 0x6f645b thru 0x6f7fb4 (0x1b5a bytes) All operations completed A-OK! PASS 16b (17s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 21:46:12 (1772678772) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (2s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 21:46:14 (1772678774) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.266363 s, 31.5 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.224377 s, 37.4 MB/s ldlm.namespaces.MGC192.168.201.111@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9c1807b0a800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9c180948d800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9c1807b0a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9c180948d800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9c1807b0a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9c180948d800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.18708 s, 16.9 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.07237 s, 50.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.55847 s, 67.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.97717 s, 26.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.76381 s, 18.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.79262 s, 37.5 MB/s PASS 16d (31s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 21:46:45 (1772678805) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.17156 s, 61.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603424 s, 17.4 MB/s PASS 16e (4s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 21:46:49 (1772678809) TIMEOUT PASS 16f (24s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 21:47:13 (1772678833) TIMEOUT PASS 16g (25s) == sanityn test 16h: mmap read after truncate file ======= 21:47:38 (1772678858) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (4s) == sanityn test 16i: read after truncate file ============ 21:47:42 (1772678862) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (4s) == sanityn test 16j: race dio with buffered i/o ========== 21:47:46 (1772678866) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.01063 s, 25.9 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.596214 s, 44.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.997047 s, 26.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.929191 s, 28.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.27133 s, 20.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.12845 s, 23.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.48267 s, 17.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.29917 s, 20.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.38557 s, 18.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.53299 s, 17.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.62809 s, 16.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.04564 s, 25.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.18729 s, 22.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.44063 s, 18.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.52871 s, 17.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.42275 s, 18.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.59948 s, 16.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.65295 s, 15.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.75904 s, 14.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (19s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 21:48:05 (1772678885) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported truncating to largest ever: 0x232e1d3 fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: skipping zero size read test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported truncating to largest ever: 0x232e1d3 fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 All operations completed A-OK! Wed Mar 4 21:48:27 EST 2026 All operations completed A-OK! All operations completed A-OK! + fsx 2 done: rc=0 Wed Mar 4 21:48:27 EST 2026 + fsx 3 done: rc=0 Wed Mar 4 21:48:27 EST 2026 + fsx 4 done: rc=0 Wed Mar 4 21:48:27 EST 2026 + fsx 5 done: rc=0 Wed Mar 4 21:48:27 EST 2026 PASS 16k (26s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 21:48:31 (1772678911) fail_loc=0x8000030a PASS 17 (7s) == sanityn test 18: mmap sanity check =========================================================================================== 21:48:38 (1772678918) mmap test1: basic mmap operation (PASS, 0.077725s) mmap test2: MAP_PRIVATE not write back (PASS, 0.063912s) mmap test3: concurrent mmap ops on two nodes (PASS, 7.0058s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.3s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.1055s) mmap test6: check mmap write/read content on two nodes (PASS, 0.11019s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 0.67674s) mmap test11: multi-thread shared mmap (PASS, 0.64286s) mmap test10: mtime not change for readonly mmap access (PASS, 1.0546s) PASS 18 (18s) == sanityn test 19: test concurrent uncached read races ========================================================================= 21:48:56 (1772678936) oleg111-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg111-client: oleg111-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (3s) == sanityn test 20: test extra readahead page left in cache ============================================================== 21:48:59 (1772678939) PASS 20 (4s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 21:49:03 (1772678943) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (3s) == sanityn test 23: others should see updated atime while another read============================================================== 21:49:06 (1772678946) now is 1772678948 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7555 new atime is 1772679009 PASS 23 (66s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 21:50:12 (1772679012) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 7168 3762176 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 33792 3735552 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 40960 7497728 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 7168 3762176 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 33792 3735552 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 40960 7497728 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.6K 387 490.2K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 402 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.3K 387 228.9K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.6K 387 490.2K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 402 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.3K 387 228.9K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 3.2M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 7.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 33.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 40.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502378 387 501991 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118034 402 117632 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117199 399 116800 1% /mnt/lustre2[OST:1] filesystem_summary: 234819 387 234432 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 7168 3762176 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 33792 3735552 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 40960 7497728 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.6K 387 490.2K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 402 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.3K 387 228.9K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502378 387 501991 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118034 402 117632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117199 399 116800 1% /mnt/lustre[OST:1] filesystem_summary: 234819 387 234432 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502378 387 501991 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118034 402 117632 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117199 399 116800 1% /mnt/lustre2[OST:1] filesystem_summary: 234819 387 234432 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 7168 3762176 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 33792 3735552 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 40960 7497728 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 7168 3762176 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 33792 3735552 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 40960 7497728 1% /mnt/lustre2 PASS 24a (4s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 21:50:16 (1772679016) PASS 24b (4s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 21:50:20 (1772679020) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (5s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 21:50:25 (1772679025) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (2s) == sanityn test 26a: allow mtime to get older ============ 21:50:27 (1772679027) utime: good mknod times 1772679027 <= 1772679028 <= 1772679028 for /mnt/lustre/f26a utime: good mknod times 1772679027 <= 1772679028 <= 1772679028 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (4s) == sanityn test 26b: sync mtime between ost and mds ====== 21:50:32 (1772679032) PASS 26b (6s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 21:50:37 (1772679037) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7555 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-03-04 21:50:38.000000000 -0500 Birth: 2026-03-04 21:50:39.000000000 -0500 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-03-04 21:50:38.000000000 -0500 Birth: 2026-03-04 21:50:39.000000000 -0500 PASS 26c (5s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 21:50:43 (1772679043) 0 PASS 30 (6s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 21:50:49 (1772679049) fail_loc=0x314 PASS 31a (5s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 21:50:54 (1772679054) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg111-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.111@tcp:/lustre 7542784 41984 7494656 1% /mnt/lustre2 PASS 31b (14s) == sanityn test 31r: open-rename(replace) race =========== 21:51:08 (1772679068) -rw-r--r-- 1 root root 0 Mar 4 21:51 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (6s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 21:51:14 (1772679074) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1207: 42849 Terminated tail -f $tfile PASS 31s (4s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 21:51:18 (1772679078) PASS 31t (4s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 21:51:23 (1772679083) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (1s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 21:51:24 (1772679084) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (2s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 21:51:26 (1772679086) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (1s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 21:51:27 (1772679087) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (2s) == sanityn test 34: no lock timeout under IO ============= 21:51:29 (1772679089) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg111-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9c1807b0a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c1807b0a800.ost_server_uuid in FULL state after 0 sec oleg111-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c1807b0a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c1807b0a800.ost_server_uuid in IDLE state after 0 sec oleg111-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9c1807b0a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c1807b0a800.ost_server_uuid in FULL state after 0 sec oleg111-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c1807b0a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c1807b0a800.ost_server_uuid in IDLE state after 0 sec oleg111-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9c1807b0a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c1807b0a800.ost_server_uuid in IDLE state after 0 sec oleg111-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c1807b0a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c1807b0a800.ost_server_uuid in IDLE state after 0 sec PASS 34 (100s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 21:53:09 (1772679189) fail_loc=0x80000317 Race attempt 0 Wait for 48896 49018 for 60 sec... - open/close 2302 (time 1772679202.95 total 10.01 last 229.95) total: 4000 open/close in 16.08 seconds: 248.73 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.201.111@tcp:/lustre 7.2G 44M 7.2G 1% /mnt/lustre 192.168.201.111@tcp:/lustre 7.2G 44M 7.2G 1% /mnt/lustre2 PASS 35 (66s) == sanityn test 36: handle ESTALE/open-unlink correctly == 21:54:16 (1772679256) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg111-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.660767 s, 79.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7555 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7493632) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.522395 s, 100 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7555 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7493632) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.539881 s, 97.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7555 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7493632) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.523658 s, 100 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7555 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7493632) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.49463 s, 106 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7555 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7493632) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.52336 s, 100 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7555 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7491584) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.500345 s, 105 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7555 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7493632) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.572019 s, 91.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7555 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7493632) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.608405 s, 86.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7555 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7491584) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.623503 s, 84.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7555 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7493632) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.532173 s, 98.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7555 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7491584) after_dd(7442432) after(7493632) PASS 36 (237s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 21:58:12 (1772679492) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7555 - create 5709 (time 1772679504.29 total 10.00 last 570.89) total: 10000 create in 19.51 seconds: 512.51 ops/second PASS 37 (25s) == sanityn test 39a: file mtime does not change after rename ========================================================== 21:58:37 (1772679517) repeat after cancel_lru_locks PASS 39a (3s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 21:58:40 (1772679520) repeat after cancel_lru_locks PASS 39b (4s) == sanityn test 39c: check truncate mtime update ================================================================================ 21:58:44 (1772679524) repeat after cancel_lru_locks PASS 39c (4s) == sanityn test 39d: sync write should update mtime ====== 21:58:48 (1772679528) fail_loc=0x411 fail_loc=0 PASS 39d (2s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 21:58:51 (1772679531) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (9s) == sanityn test 40c: pdirops: link and others ======================================================================== 21:59:00 (1772679540) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (9s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 21:59:09 (1772679549) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (8s) == sanityn test 40e: pdirops: rename and others ======================================================================== 21:59:17 (1772679557) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (8s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 21:59:25 (1772679565) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (6s) == sanityn test 41b: pdirops: create vs create ======================================================================== 21:59:31 (1772679571) open: File exists Conflict PASS 41b (5s) == sanityn test 41c: pdirops: create vs link ======================================================================== 21:59:37 (1772679577) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (5s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 21:59:42 (1772679582) Conflict PASS 41d (6s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 21:59:48 (1772679588) Conflict PASS 41e (5s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 21:59:53 (1772679593) Conflict PASS 41f (6s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 21:59:59 (1772679599) Conflict PASS 41g (5s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 22:00:04 (1772679604) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2318: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (5s) == sanityn test 41i: reint_open: create vs create ======== 22:00:09 (1772679609) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (727s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 22:12:16 (1772680336) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (6s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 22:12:22 (1772680342) open: File exists Conflict PASS 42b (6s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 22:12:28 (1772680348) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (5s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 22:12:33 (1772680353) Conflict PASS 42d (6s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 22:12:39 (1772680359) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (5s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 22:12:44 (1772680364) Conflict PASS 42f (6s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 22:12:50 (1772680370) Conflict PASS 42g (5s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 22:12:55 (1772680375) Conflict PASS 42h (5s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 22:13:00 (1772680380) PASS 43a (19s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 22:13:19 (1772680399) Conflict PASS 43b (5s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 22:13:24 (1772680404) Conflict PASS 43c (5s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 22:13:29 (1772680409) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (5s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 22:13:34 (1772680414) Conflict PASS 43e (5s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 22:13:39 (1772680419) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (5s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 22:13:44 (1772680424) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (5s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 22:13:49 (1772680429) Conflict PASS 43h (5s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 22:13:54 (1772680434) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (2s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 22:13:56 (1772680436) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists PASS 43j (37s) == sanityn test 43k: unlink vs create ==================== 22:14:33 (1772680473) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (422s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 22:21:35 (1772680895) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Wed Mar 4 22:21:39 EST 2026 PASS 44a (6s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 22:21:41 (1772680901) open: File exists Conflict PASS 44b (5s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 22:21:46 (1772680906) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (5s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 22:21:51 (1772680911) Conflict PASS 44d (5s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 22:21:56 (1772680916) Conflict PASS 44e (6s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 22:22:02 (1772680922) Conflict PASS 44f (5s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 22:22:07 (1772680927) Conflict PASS 44g (5s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 22:22:12 (1772680932) Conflict PASS 44h (6s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 22:22:18 (1772680938) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (1s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 22:22:19 (1772680939) PASS 45a (28s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 22:22:47 (1772680967) Conflict PASS 45b (5s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 22:22:52 (1772680972) Conflict PASS 45c (5s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 22:22:57 (1772680977) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (6s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 22:23:03 (1772680983) Conflict PASS 45e (5s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 22:23:08 (1772680988) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (5s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 22:23:13 (1772680993) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (6s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 22:23:19 (1772680999) Conflict PASS 45h (4s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 22:23:23 (1772681003) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (1s) == sanityn test 45j: read vs rename ====================== 22:23:25 (1772681005) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (427s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 22:30:32 (1772681432) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (5s) == sanityn test 46b: pdirops: link vs create ======================================================================== 22:30:37 (1772681437) open: File exists Conflict PASS 46b (5s) == sanityn test 46c: pdirops: link vs link ======================================================================== 22:30:42 (1772681442) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (5s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 22:30:47 (1772681447) Conflict PASS 46d (6s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 22:30:53 (1772681453) Conflict PASS 46e (5s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 22:30:58 (1772681458) Conflict PASS 46f (5s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 22:31:03 (1772681463) Conflict PASS 46g (5s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 22:31:09 (1772681469) Conflict PASS 46h (6s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 22:31:14 (1772681474) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (1s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 22:31:15 (1772681475) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (1s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 22:31:16 (1772681476) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (1s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 22:31:17 (1772681477) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (1s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 22:31:18 (1772681478) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (1s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 22:31:19 (1772681479) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (1s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 22:31:20 (1772681480) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (1s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 22:31:21 (1772681481) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (1s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 22:31:22 (1772681482) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.00648492 s, 1.6 MB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (8s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 22:31:30 (1772681490) 0+1 records in 0+1 records out 158 bytes copied, 0.00365247 s, 43.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3581: kill: (216620) - No such process PASS 51a (4s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 22:31:34 (1772681494) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00295225 s, 347 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192073 s, 54.6 MB/s 3145728 PASS 51b (15s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 22:31:49 (1772681509) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00457554 s, 224 kB/s PASS 51c (6s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 22:31:55 (1772681515) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148353 s, 70.7 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (3s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 22:31:58 (1772681518) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2453 0x995 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2453 0x995 0x240000400 read lease(1) has applied. PASS 51e (4s) == sanityn test 54: rename locking ======================= 22:32:02 (1772681522) llite.lustre-ffff9c1807b0a800.xattr_cache=1 llite.lustre-ffff9c180948d800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505011 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-04 22:32:03.000000000 -0500 Modify: 2026-03-04 22:32:03.000000000 -0500 Change: 2026-03-04 22:32:03.000000000 -0500 Birth: 2026-03-04 22:32:03.000000000 -0500 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505014 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-04 22:32:09.000000000 -0500 Modify: 2026-03-04 22:32:09.000000000 -0500 Change: 2026-03-04 22:32:09.000000000 -0500 Birth: 2026-03-04 22:32:09.000000000 -0500 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505017 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-04 22:32:14.000000000 -0500 Modify: 2026-03-04 22:32:14.000000000 -0500 Change: 2026-03-04 22:32:14.000000000 -0500 Birth: 2026-03-04 22:32:15.000000000 -0500 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505016 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-04 22:32:14.000000000 -0500 Modify: 2026-03-04 22:32:15.000000000 -0500 Change: 2026-03-04 22:32:15.000000000 -0500 Birth: 2026-03-04 22:32:15.000000000 -0500 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505019 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-04 22:32:20.000000000 -0500 Modify: 2026-03-04 22:32:20.000000000 -0500 Change: 2026-03-04 22:32:20.000000000 -0500 Birth: 2026-03-04 22:32:20.000000000 -0500 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-04 22:32:20.000000000 -0500 Modify: 2026-03-04 22:32:20.000000000 -0500 Change: 2026-03-04 22:32:20.000000000 -0500 Birth: 2026-03-04 22:32:20.000000000 -0500 PASS 54 (25s) == sanityn test 55a: rename vs unlink target dir ========= 22:32:27 (1772681547) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (8s) == sanityn test 55b: rename vs unlink source dir ========= 22:32:35 (1772681555) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (7s) == sanityn test 55c: rename vs unlink orphan target dir == 22:32:42 (1772681562) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (13s) == sanityn test 55d: rename file vs link ================= 22:32:55 (1772681575) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (10s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 22:33:05 (1772681585) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (1s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 22:33:06 (1772681586) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (13s) == sanityn test 55g: rename: race with trylock =========== 22:33:19 (1772681599) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (14s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 22:33:33 (1772681613) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1772681614 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 2.34766 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (29s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 22:34:02 (1772681642) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1772681644 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 3.48783 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (40s) == sanityn test 60: Verify data_version behaviour ======== 22:34:42 (1772681682) PASS 60 (2s) == sanityn test 70a: cd directory && rm directory ======== 22:34:44 (1772681684) PASS 70a (3s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 22:34:47 (1772681687) PASS 70b (3s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 22:34:50 (1772681690) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00261326 s, 25.1 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (1s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 22:34:51 (1772681691) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0035204 s, 18.6 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (1s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 22:34:53 (1772681693) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (1s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 22:34:54 (1772681694) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0050069 s, 13.1 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (1s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 22:34:55 (1772681695) llite.lustre-ffff9c1807b0a800.xattr_cache=1 llite.lustre-ffff9c180948d800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (2s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 22:34:57 (1772681697) llite.lustre-ffff9c1807b0a800.xattr_cache=1 llite.lustre-ffff9c180948d800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (2s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 22:34:59 (1772681699) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00269377 s, 380 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00266931 s, 384 kB/s 229379: get lock2 229381: get lock1 229379: done 229379: get lock1 229381: done 229381: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 229381: cannot get lock2: Resource deadlock avoided 229381: exit rc=0 229379: done 229379: exit rc=0 PASS 74 (6s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 22:35:05 (1772681705) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0205205 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00949584 s, 110 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4264: 229962 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (7s) == sanityn test 76: Verify MDT open_files listing ======== 22:35:12 (1772681712) opening files: ........ closing files: ........ PASS 76 (20s) == sanityn test 77a: check FIFO NRS policy =============== 22:35:32 (1772681732) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.145681 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442908 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676618 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118391 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106088 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949605 s, 11.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130021 s, 8.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126289 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120784 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127181 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147496 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144737 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123407 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137017 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140143 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122204 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134134 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202477 s, 51.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206419 s, 50.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184081 s, 57.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294749 s, 35.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245445 s, 42.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141381 s, 74.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234285 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410548 s, 25.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288667 s, 36.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027927 s, 37.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040073 s, 26.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417434 s, 25.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385737 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035077 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029641 s, 35.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341624 s, 30.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1807b0a800.lock_unused_count=1 removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg111-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg111-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (3s) == sanityn test 77b: check CRR-N NRS policy ============== 22:35:35 (1772681735) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.156759 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037651 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465804 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691973 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614528 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568994 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635161 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641953 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616213 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844545 s, 12.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9c1807b0a800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104722 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10884 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781771 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131783 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143995 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146571 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00861851 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015754 s, 66.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224059 s, 46.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160244 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265295 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265563 s, 39.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128014 s, 81.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131663 s, 79.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160816 s, 65.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235485 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528113 s, 19.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272432 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283094 s, 37.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319818 s, 32.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233692 s, 44.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339624 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155901 s, 67.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1807b0a800.lock_unused_count=1 removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg111-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg111-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.153597 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468756 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676527 s, 15.5 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0001-osc-ffff9c1807b0a800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761587 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084422 s, 12.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117128 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126683 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146946 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137854 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145349 s, 7.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1807b0a800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9c1807b0a800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144679 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151048 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179248 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160764 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161535 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159709 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178418 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214113 s, 49.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232251 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198158 s, 52.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023686 s, 44.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017329 s, 60.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015937 s, 65.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140191 s, 74.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213183 s, 49.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106189 s, 98.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00876296 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253148 s, 41.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00587682 s, 178 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316703 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021143 s, 49.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182386 s, 57.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00679978 s, 154 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg111-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg111-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (5s) == sanityn test 77c: check ORR NRS policy ================ 22:35:40 (1772681740) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.142183 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372053 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223722 s, 46.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720539 s, 14.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971041 s, 10.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105741 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938989 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075133 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885546 s, 11.8 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0001-osc-ffff9c1807b0a800.lock_unused_count=5 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108593 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108045 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100576 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113762 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100023 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134422 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118358 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125751 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159526 s, 65.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125474 s, 83.6 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196819 s, 53.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146235 s, 71.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240675 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00963348 s, 109 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265565 s, 39.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1807b0a800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024676 s, 42.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114467 s, 91.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127048 s, 82.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115712 s, 90.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150191 s, 69.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191584 s, 54.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102949 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169515 s, 61.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229902 s, 45.6 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg111-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg111-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.147974 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404649 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455196 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736823 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895851 s, 11.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9c1807b0a800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149776 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153437 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129039 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143882 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135471 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140563 s, 7.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140004 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148779 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129749 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163773 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184086 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178846 s, 5.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018702 s, 56.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222264 s, 47.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022482 s, 46.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231994 s, 45.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1807b0a800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9c1807b0a800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226461 s, 46.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233675 s, 44.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286413 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302099 s, 34.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175175 s, 59.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247955 s, 42.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00815427 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166261 s, 63.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00948638 s, 111 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344666 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120136 s, 87.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232991 s, 45.0 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg111-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg111-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.13833 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414259 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451079 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685312 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059562 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04478 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082847 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07898 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656166 s, 16.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954797 s, 11.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9c1807b0a800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151721 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306641 s, 34.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035493 s, 29.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162799 s, 64.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418572 s, 25.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493224 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037046 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177468 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328597 s, 31.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460511 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170398 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568503 s, 18.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197076 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201263 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00926693 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268899 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141473 s, 74.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169602 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01647 s, 63.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369622 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527933 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.007913 s, 133 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183319 s, 5.7 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg111-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg111-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (5s) == sanityn test 77d: check TRR nrs policy ================ 22:35:45 (1772681745) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.212629 s, 78.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381734 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612283 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636871 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121692 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157001 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143823 s, 7.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1807b0a800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16405 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165883 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149337 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157023 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159668 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166849 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165858 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148166 s, 7.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182143 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147633 s, 7.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307356 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281543 s, 37.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040458 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301026 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158938 s, 66.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277788 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169018 s, 62.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00912217 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012267 s, 85.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406114 s, 25.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238035 s, 44.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100137 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207705 s, 50.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215293 s, 48.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00791848 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044479 s, 23.6 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg111-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg111-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.136576 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417356 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478251 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515569 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617042 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711681 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918271 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938724 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101395 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936571 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111965 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107085 s, 9.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9c1807b0a800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112803 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123947 s, 8.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953035 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107521 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11616 s, 9.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238497 s, 44.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224401 s, 46.7 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256962 s, 40.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243935 s, 43.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262261 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224662 s, 46.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244927 s, 42.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341377 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135292 s, 77.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192176 s, 54.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00916047 s, 114 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277257 s, 37.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019539 s, 53.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240523 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190694 s, 55.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017262 s, 60.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1807b0a800.lock_unused_count=1 removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg111-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg111-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.17276 s, 97.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238164 s, 44.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104871 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103749 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128702 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109125 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12717 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145097 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12187 s, 8.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126891 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132166 s, 7.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157355 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168778 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144964 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132625 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142888 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144005 s, 7.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218071 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235436 s, 44.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117928 s, 88.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273675 s, 38.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278036 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392594 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163535 s, 64.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020295 s, 51.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160794 s, 65.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161657 s, 64.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266939 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256193 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123286 s, 85.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185549 s, 56.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1807b0a800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193746 s, 54.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116154 s, 90.3 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg111-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg111-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (6s) == sanityn test 77e: check TBF NID nrs policy ============ 22:35:51 (1772681751) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.201.11@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.193332 s, 86.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429368 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657243 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948239 s, 11.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144435 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126794 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129447 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136983 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127818 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127243 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154497 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133465 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175906 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164388 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144039 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181597 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185935 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305629 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294552 s, 35.6 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff9c1807b0a800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9c1807b0a800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039655 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351414 s, 29.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308192 s, 34.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270993 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184175 s, 56.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180078 s, 58.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285364 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422482 s, 24.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480566 s, 21.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133114 s, 78.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380004 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190374 s, 55.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208819 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277977 s, 37.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg111-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg111-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.129355 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309142 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481559 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391618 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541219 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670862 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617876 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062141 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812112 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921943 s, 11.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1807b0a800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922729 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11687 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113194 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995749 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107215 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116045 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110662 s, 9.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1807b0a800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742273 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851646 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740112 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753589 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784719 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811986 s, 12.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858756 s, 12.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812495 s, 12.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820965 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770184 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844578 s, 12.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945059 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.096075 s, 10.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862981 s, 12.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879728 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878539 s, 11.9 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg111-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg111-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.132835 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774703 s, 13.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925063 s, 11.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848677 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126777 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131061 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132812 s, 7.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189308 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163466 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163127 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150212 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145806 s, 7.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163264 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147637 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157389 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20762 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213475 s, 4.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1807b0a800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9c1807b0a800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152522 s, 68.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221913 s, 47.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187049 s, 56.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244608 s, 42.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252439 s, 41.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030527 s, 34.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394806 s, 26.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03575 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319084 s, 32.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00880223 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292596 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160313 s, 65.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013277 s, 79.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230325 s, 45.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287498 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187661 s, 55.9 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg111-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg111-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.159174 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023692 s, 44.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320273 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770477 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808515 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794413 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945973 s, 11.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1807b0a800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9c1807b0a800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104022 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852024 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106344 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105605 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959449 s, 10.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886206 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112786 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107858 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109664 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11554 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110779 s, 94.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131395 s, 79.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff9c1807b0a800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9c1807b0a800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163197 s, 64.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201857 s, 51.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301491 s, 34.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100516 s, 104 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226602 s, 46.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173866 s, 60.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229362 s, 45.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00950183 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00863532 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010285 s, 102 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160668 s, 65.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195976 s, 53.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200603 s, 52.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00513589 s, 204 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg111-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg111-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (8s) == sanityn test 77f: check TBF JobID nrs policy ========== 22:35:59 (1772681759) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg111-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.127165 s, 132 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252122 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357898 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070158 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644048 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098925 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798603 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089985 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119673 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970248 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118957 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136085 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114361 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142613 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128855 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108539 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105764 s, 9.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169261 s, 62.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00803694 s, 130 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256782 s, 40.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255838 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162819 s, 64.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020278 s, 51.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200202 s, 52.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250946 s, 41.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00642002 s, 163 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140888 s, 74.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00738271 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307207 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013848 s, 75.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00504276 s, 208 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03424 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00493218 s, 213 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg111-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg111-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg111-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.199178 s, 84.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267055 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584685 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490881 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132476 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142665 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113476 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107978 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147954 s, 7.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138336 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158286 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152608 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148105 s, 7.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1349 s, 7.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148563 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14637 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153884 s, 6.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1807b0a800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9c1807b0a800.lock_unused_count=2 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185918 s, 56.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145724 s, 72.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208385 s, 50.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187453 s, 55.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217925 s, 48.1 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff9c1807b0a800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133315 s, 78.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00980169 s, 107 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146593 s, 71.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237686 s, 44.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154522 s, 67.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032267 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162207 s, 64.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00785887 s, 133 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239768 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190228 s, 55.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00848202 s, 124 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg111-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg111-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg111-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.116988 s, 143 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652829 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071879 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091194 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100166 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109422 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094097 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113092 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875145 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996199 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115779 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098003 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990587 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102302 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957943 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10746 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115361 s, 9.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1807b0a800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9c1807b0a800.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014201 s, 73.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184518 s, 56.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166126 s, 63.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201976 s, 51.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128322 s, 81.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181615 s, 57.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180937 s, 58.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199645 s, 52.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155759 s, 67.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115424 s, 90.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00401901 s, 261 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101119 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025492 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0053852 s, 195 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00371614 s, 282 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010788 s, 97.2 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg111-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg111-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg111-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.14944 s, 112 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370776 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043353 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126115 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113213 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154828 s, 6.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161424 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150908 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125504 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163839 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147039 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171178 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166075 s, 6.3 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff9c1807b0a800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9c1807b0a800.lock_unused_count=2 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166238 s, 6.3 MB/s execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188308 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174925 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177234 s, 5.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172175 s, 60.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209122 s, 50.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264283 s, 39.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026422 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191984 s, 54.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00587457 s, 178 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135896 s, 77.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313529 s, 33.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161568 s, 64.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127888 s, 82.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101533 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00719742 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431025 s, 24.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202825 s, 51.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423746 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146956 s, 71.4 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg111-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg111-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (7s) == sanityn test 77g: Change TBF type directly ============ 22:36:06 (1772681766) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (4s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 22:36:10 (1772681770) oleg111-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg111-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg111-client: oleg111-server: ssh exited with exit code 2 oleg111-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg111-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg111-client: oleg111-server: ssh exited with exit code 22 oleg111-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg111-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg111-client: oleg111-server: ssh exited with exit code 22 oleg111-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg111-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg111-client: oleg111-server: ssh exited with exit code 22 PASS 77h (4s) == sanityn test 77i: Change rank of TBF rule ============= 22:36:14 (1772681774) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (7s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 22:36:21 (1772681781) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.14904 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764168 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841577 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738573 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755037 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775045 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958982 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952269 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077166 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688707 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112661 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143771 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156997 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126205 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152754 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149573 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166632 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247111 s, 42.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227686 s, 46.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244643 s, 42.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270007 s, 38.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177308 s, 59.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026236 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242417 s, 43.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198946 s, 52.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278734 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111943 s, 93.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00981345 s, 107 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203476 s, 51.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157768 s, 66.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158043 s, 66.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413065 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124317 s, 84.3 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg111-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg111-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.95807 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4358 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (49s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 22:37:10 (1772681830) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg111-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg111-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg111-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg111-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg111-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.22268 s, 13.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159684 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159677 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15739 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139564 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154973 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175443 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185996 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15909 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183675 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179702 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177614 s, 5.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166632 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18333 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184018 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178358 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180208 s, 5.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233756 s, 44.9 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228594 s, 45.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219816 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257348 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171734 s, 61.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248749 s, 42.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267357 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253238 s, 41.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232262 s, 45.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346338 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00861962 s, 122 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282309 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00595668 s, 176 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00485323 s, 216 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0047313 s, 222 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00481133 s, 218 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg111-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg111-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8322 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.953 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg111-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg111-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg111-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg111-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg111-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.22655 s, 13.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116164 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120291 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121182 s, 8.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1551 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135256 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140211 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121274 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125261 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150307 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141561 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163653 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175105 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182374 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181857 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165656 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163741 s, 6.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257892 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218492 s, 48.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268682 s, 39.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021474 s, 48.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266375 s, 39.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174517 s, 60.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294967 s, 35.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314213 s, 33.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210924 s, 49.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275586 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304227 s, 34.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269834 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211396 s, 49.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259686 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262471 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033037 s, 31.7 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg111-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg111-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8322 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9608 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (137s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 22:39:27 (1772681967) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg111-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg111-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg111-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg111-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg111-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.22671 s, 13.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824715 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627965 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803839 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616364 s, 17.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853101 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833579 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899358 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820371 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161546 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161908 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173859 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183794 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176888 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153711 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176666 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163071 s, 6.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184238 s, 56.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196348 s, 53.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211178 s, 49.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236556 s, 44.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205182 s, 51.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154836 s, 67.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022692 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200736 s, 52.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165161 s, 63.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244288 s, 42.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248003 s, 42.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00816529 s, 128 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00476816 s, 220 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00886466 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107667 s, 97.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00536734 s, 195 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg111-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg111-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7314 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9546 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg111-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg111-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg111-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg111-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg111-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.23692 s, 13.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119247 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132465 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121195 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122773 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122235 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120648 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135426 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150906 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145784 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124585 s, 8.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142123 s, 7.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122084 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13364 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142085 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146887 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15753 s, 6.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232184 s, 45.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292666 s, 35.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244674 s, 42.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027073 s, 38.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142644 s, 73.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026962 s, 38.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143122 s, 73.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187544 s, 55.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149943 s, 69.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270186 s, 38.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024319 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300853 s, 34.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193024 s, 54.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329577 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136379 s, 76.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308638 s, 34.0 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg111-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg111-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8563 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9019 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (130s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 22:41:37 (1772682097) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg111-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.271861 s, 61.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326285 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720581 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957257 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809761 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907033 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982154 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886075 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949751 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897657 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823061 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937249 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893977 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111045 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086736 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090155 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107963 s, 9.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138168 s, 75.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154843 s, 67.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196147 s, 53.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214188 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198667 s, 52.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197041 s, 53.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203372 s, 51.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104945 s, 99.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230758 s, 45.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00779457 s, 135 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194149 s, 54.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233639 s, 44.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00947466 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00570332 s, 184 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0039415 s, 266 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00504052 s, 208 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg111-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg111-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94278 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7351 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.201.11@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.201.11@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.241808 s, 69.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461423 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434871 s, 24.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130774 s, 8.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121411 s, 8.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11564 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122846 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119632 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130533 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136387 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131417 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127778 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118711 s, 8.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227962 s, 4.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243489 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307234 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291533 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214257 s, 48.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292674 s, 35.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347163 s, 30.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322763 s, 32.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303443 s, 34.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037988 s, 27.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312915 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01267 s, 82.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166969 s, 62.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314311 s, 33.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356475 s, 29.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151534 s, 69.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205184 s, 51.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103534 s, 101 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125898 s, 83.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327477 s, 32.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg111-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg111-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.95114 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73978 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.201.11@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg111-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.246832 s, 68.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280727 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296216 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993469 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803474 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101271 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887333 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890378 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113026 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107744 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087477 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969251 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104164 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110581 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115913 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121607 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113173 s, 9.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146192 s, 71.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170424 s, 61.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01527 s, 68.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00912472 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189566 s, 55.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162398 s, 64.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219169 s, 47.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00746857 s, 140 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183103 s, 57.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266954 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317311 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235331 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00361242 s, 290 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00648755 s, 162 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0036887 s, 284 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00503782 s, 208 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg111-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg111-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9284 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.87842 s, 21.5 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg111-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.445075 s, 37.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409429 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318129 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905947 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825671 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116025 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088735 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971383 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11748 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119371 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127697 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107186 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120909 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143641 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136711 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160398 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180052 s, 5.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264267 s, 39.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196516 s, 53.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244421 s, 42.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222136 s, 47.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152237 s, 68.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194134 s, 54.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233311 s, 44.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254937 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197556 s, 53.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266923 s, 39.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235291 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285537 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185341 s, 56.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024151 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00354982 s, 295 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00492171 s, 213 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg111-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg111-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.78888 s, 10.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73924 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94747 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74104 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg111-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.886517 s, 18.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370325 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612069 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839013 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685294 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567654 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835946 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712615 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676406 s, 15.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116923 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117043 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132419 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136112 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143902 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174353 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171579 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15236 s, 6.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208107 s, 50.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103034 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138715 s, 75.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158797 s, 66.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163797 s, 64.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00844906 s, 124 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179437 s, 58.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160959 s, 65.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214677 s, 48.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216662 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00421277 s, 249 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221637 s, 47.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00682874 s, 154 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00486278 s, 216 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255201 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00219046 s, 479 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg111-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg111-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6962 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4394 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg111-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.27459 s, 61.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433455 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402337 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114225 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126459 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16329 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143804 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157613 s, 6.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162664 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146686 s, 7.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147464 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150062 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15199 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16517 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168877 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186259 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172611 s, 6.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206975 s, 50.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262611 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302557 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282095 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306325 s, 34.2 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278686 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254313 s, 41.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388113 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342649 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186458 s, 56.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00940305 s, 112 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453736 s, 23.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00891086 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0086123 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00474706 s, 221 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00684991 s, 153 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg111-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg111-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.95637 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73732 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg111-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.499669 s, 33.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479625 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364738 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141694 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136842 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153146 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161691 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15379 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150067 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192928 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171007 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188417 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174454 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191502 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21492 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200461 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192133 s, 5.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207831 s, 50.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00952574 s, 110 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100603 s, 104 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174444 s, 60.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234329 s, 44.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199407 s, 52.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212202 s, 49.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244993 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207313 s, 50.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179581 s, 58.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226993 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227272 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207546 s, 50.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025873 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00419165 s, 250 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0036729 s, 285 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg111-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg111-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.80493 s, 10.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74363 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.78192 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73253 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (359s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 22:47:36 (1772682456) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg111-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.293883 s, 57.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359549 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443304 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822585 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635279 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810056 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739142 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897711 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865536 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817299 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733103 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868398 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542438 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748779 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856645 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652046 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472643 s, 22.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00821792 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194779 s, 53.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029865 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00674096 s, 156 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139537 s, 75.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116312 s, 90.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158308 s, 66.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00215553 s, 486 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00218853 s, 479 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00511038 s, 205 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00658029 s, 159 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00772653 s, 136 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00703757 s, 149 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00906892 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00494697 s, 212 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00523877 s, 200 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg111-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg111-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.99083 s, 21.0 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76335 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (41s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 22:48:17 (1772682497) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.201.11@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.201.11@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.28596 s, 58.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316078 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322722 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985593 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102925 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848972 s, 12.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834861 s, 12.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890146 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946284 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883977 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103352 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103907 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879149 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232019 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249504 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279698 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309719 s, 3.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00415316 s, 252 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141499 s, 74.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162194 s, 64.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195163 s, 53.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174798 s, 60.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170811 s, 61.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160572 s, 65.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207839 s, 50.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197981 s, 53.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00432403 s, 242 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115441 s, 90.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00468615 s, 224 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0043825 s, 239 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00437309 s, 240 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00389897 s, 269 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00547587 s, 191 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg111-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg111-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94153 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73458 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (40s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 22:48:57 (1772682537) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.201.11@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg111-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.255187 s, 65.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264583 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272214 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485244 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685792 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606286 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801228 s, 13.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691787 s, 15.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076945 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965231 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087331 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880302 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837819 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762141 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107662 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875206 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105896 s, 9.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164243 s, 63.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169186 s, 62.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184041 s, 57.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016293 s, 64.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136813 s, 76.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188598 s, 55.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170635 s, 61.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00703011 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00506976 s, 207 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217157 s, 48.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210272 s, 49.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00705174 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00305859 s, 343 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020689 s, 50.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263209 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00807513 s, 130 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg111-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg111-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94125 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97293 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (32s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 22:49:29 (1772682569) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg111-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.457736 s, 36.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024059 s, 43.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428605 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375601 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333612 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394636 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583919 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062911 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483954 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394948 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811146 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114623 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110243 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364142 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.36085 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.405764 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.403033 s, 2.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191839 s, 54.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154356 s, 67.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145757 s, 71.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142152 s, 73.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167395 s, 62.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205148 s, 51.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154153 s, 68.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208718 s, 50.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167771 s, 62.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206122 s, 50.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159106 s, 65.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181524 s, 57.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00351874 s, 298 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240605 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00347476 s, 302 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00567384 s, 185 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg111-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg111-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77596 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.75689 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.92383 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73829 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (81s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 22:50:50 (1772682650) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg111-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.851177 s, 19.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203219 s, 51.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249724 s, 42.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422884 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325145 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486696 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634299 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600252 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655831 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163168 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175871 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187153 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180563 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20643 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185524 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196862 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211936 s, 4.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019369 s, 54.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186778 s, 56.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179885 s, 58.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180016 s, 58.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160799 s, 65.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178091 s, 58.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212594 s, 49.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165249 s, 63.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154543 s, 67.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231642 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236397 s, 44.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021698 s, 48.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209693 s, 50.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00613355 s, 171 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00580961 s, 180 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00266595 s, 393 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg111-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg111-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6693 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9637 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (67s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 22:51:57 (1772682717) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg111-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.267445 s, 62.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03126 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605536 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460533 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597057 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121336 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150472 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145753 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137643 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131028 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136371 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151711 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146822 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139122 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167033 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178258 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142017 s, 7.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105397 s, 99.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00528981 s, 198 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139044 s, 75.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026793 s, 39.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155283 s, 67.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00485331 s, 216 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00488435 s, 215 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00481575 s, 218 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00340795 s, 308 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344833 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262809 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220376 s, 47.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336735 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022664 s, 46.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00496752 s, 211 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00356459 s, 294 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg111-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg111-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96533 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7453 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg111-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.503859 s, 33.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887468 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971483 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109114 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138365 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15001 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156946 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131696 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133863 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136999 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147156 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125088 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151876 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148584 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149744 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273533 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267131 s, 3.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169676 s, 61.8 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253054 s, 41.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310596 s, 33.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281338 s, 37.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422701 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308669 s, 34.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372842 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345535 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299731 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036884 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378733 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386953 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390549 s, 26.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363223 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396396 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413685 s, 25.3 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg111-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg111-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89909 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77702 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.78468 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.75522 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (123s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 22:54:00 (1772682840) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg111-client.virtnet /mnt/lustre (opts:) Stopping client oleg111-client.virtnet /mnt/lustre opts: Stopping clients: oleg111-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg111-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg111-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg111-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg111-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg111-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.11,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg111-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.11,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg111-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.11,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/60848/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.11,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg111-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg111-server: oleg111-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg111-client: oleg111-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg111-server: oleg111-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg111-client: oleg111-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg111-server: oleg111-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg111-client: oleg111-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Started clients oleg111-client.virtnet: 192.168.201.111@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 Starting client oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 Started clients oleg111-client.virtnet: 192.168.201.111@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9c180325a000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9c1807d63000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c180325a000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c1807d63000.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.244333 s, 68.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405052 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920274 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886596 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878755 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116279 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101709 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107696 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109353 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854606 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100644 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119841 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12829 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234193 s, 4.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232302 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296694 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305929 s, 3.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116917 s, 89.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312755 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015648 s, 67.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247452 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155032 s, 67.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241072 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213891 s, 49.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226911 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477114 s, 22.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324254 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233078 s, 45.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519914 s, 20.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243323 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00730929 s, 143 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310816 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056091 s, 18.7 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg111-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg111-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.92589 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9746 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.317725 s, 52.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369223 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356815 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616649 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107711 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103371 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883394 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980846 s, 10.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115711 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120752 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116077 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132265 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126267 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232485 s, 4.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220532 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259761 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260644 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235075 s, 44.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209148 s, 50.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242234 s, 43.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023477 s, 44.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021525 s, 48.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236719 s, 44.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178438 s, 58.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220584 s, 47.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00568995 s, 184 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239953 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115016 s, 91.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162789 s, 64.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00957964 s, 109 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00506035 s, 207 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00842248 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334363 s, 31.4 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg111-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg111-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.00296 s, 21.0 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76365 s, 10.7 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (144s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 22:56:24 (1772682984) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg111-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg111-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg111-client: oleg111-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (9s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 22:56:33 (1772682993) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (4s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 22:56:37 (1772682997) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg111-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.43572 s, 305 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg111-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0259 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (52s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 22:57:29 (1772683049) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.215175 s, 78.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784076 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816251 s, 12.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917311 s, 11.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788231 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934504 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710704 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114057 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962319 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955232 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107345 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104573 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12311 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950963 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113651 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115099 s, 9.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1807d63000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9c1807d63000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896428 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308003 s, 34.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182843 s, 57.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239722 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203199 s, 51.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121545 s, 86.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342186 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356289 s, 29.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365332 s, 28.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271622 s, 38.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241451 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273481 s, 38.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035675 s, 29.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031679 s, 33.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378768 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00568916 s, 184 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344482 s, 30.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9c1807d63000.lock_unused_count=1 removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg111-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg111-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97475 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.98242 s, 21.0 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.260771 s, 64.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355283 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689811 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552871 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810912 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635996 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907359 s, 11.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779658 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961049 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710309 s, 14.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101217 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693109 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814665 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09097 s, 11.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976173 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105619 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118317 s, 8.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241318 s, 43.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250254 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274388 s, 38.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026658 s, 39.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024308 s, 43.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023001 s, 45.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039841 s, 26.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137507 s, 76.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331199 s, 31.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220561 s, 47.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357591 s, 29.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359349 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351544 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00565714 s, 185 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00638819 s, 164 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00419884 s, 250 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg111-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg111-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93903 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97647 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (67s) == sanityn test 77o: Changing rank should not panic ====== 22:58:36 (1772683116) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (5s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 22:58:41 (1772683121) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (45s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 22:59:26 (1772683166) mds.MDS.mdt.nrs_policies=tbf oleg111-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg111-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg111-client: oleg111-server: ssh exited with exit code 36 oleg111-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg111-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg111-client: oleg111-server: ssh exited with exit code 22 oleg111-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg111-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg111-client: oleg111-server: ssh exited with exit code 22 oleg111-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg111-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg111-client: oleg111-server: ssh exited with exit code 22 oleg111-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg111-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg111-client: oleg111-server: ssh exited with exit code 22 oleg111-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg111-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg111-client: oleg111-server: ssh exited with exit code 22 oleg111-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg111-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg111-client: oleg111-server: ssh exited with exit code 22 oleg111-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg111-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg111-client: oleg111-server: ssh exited with exit code 22 oleg111-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg111-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg111-client: oleg111-server: ssh exited with exit code 22 oleg111-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg111-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg111-client: oleg111-server: ssh exited with exit code 22 oleg111-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg111-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg111-client: oleg111-server: ssh exited with exit code 22 oleg111-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg111-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg111-client: oleg111-server: ssh exited with exit code 22 oleg111-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg111-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg111-client: oleg111-server: ssh exited with exit code 22 oleg111-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg111-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg111-client: oleg111-server: ssh exited with exit code 22 oleg111-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg111-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg111-client: oleg111-server: ssh exited with exit code 22 oleg111-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg111-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg111-client: oleg111-server: ssh exited with exit code 22 oleg111-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg111-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg111-client: oleg111-server: ssh exited with exit code 22 oleg111-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg111-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg111-client: oleg111-server: ssh exited with exit code 22 oleg111-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg111-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg111-client: oleg111-server: ssh exited with exit code 22 oleg111-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg111-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg111-client: oleg111-server: ssh exited with exit code 22 oleg111-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg111-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg111-client: oleg111-server: ssh exited with exit code 22 oleg111-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg111-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg111-client: oleg111-server: ssh exited with exit code 22 oleg111-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg111-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg111-client: oleg111-server: ssh exited with exit code 22 oleg111-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg111-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg111-client: oleg111-server: ssh exited with exit code 22 oleg111-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg111-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg111-client: oleg111-server: ssh exited with exit code 22 oleg111-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg111-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg111-client: oleg111-server: ssh exited with exit code 22 oleg111-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg111-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg111-client: oleg111-server: ssh exited with exit code 22 oleg111-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg111-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg111-client: oleg111-server: ssh exited with exit code 22 oleg111-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg111-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg111-client: oleg111-server: ssh exited with exit code 22 oleg111-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg111-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg111-client: oleg111-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (14s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 22:59:40 (1772683180) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 270379 kill md_thread xargs PASS 77r (44s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 23:00:24 (1772683224) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (3s) == sanityn test 79: xattr: intent error ================== 23:00:27 (1772683227) File: /mnt/lustre/d79.sanityn Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935203 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-04 23:00:28.000000000 -0500 Modify: 2026-03-04 23:00:28.000000000 -0500 Change: 2026-03-04 23:00:28.000000000 -0500 Birth: 2026-03-04 23:00:28.000000000 -0500 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (13s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 23:00:40 (1772683240) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (2s) == sanityn test 80b: Accessing directory during migration ========================================================== 23:00:42 (1772683242) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (1s) == sanityn test 81a: rename and stat under striped directory ========================================================== 23:00:43 (1772683243) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (1s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 23:00:44 (1772683244) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (2s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 23:00:46 (1772683246) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (1s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 23:00:47 (1772683247) total: 2500 open/close in 6.92 seconds: 361.15 ops/second snapshot_time 1772683277.321914477 secs.nsecs start_time 1772683249.300432609 secs.nsecs elapsed_time 28.021481868 secs.nsecs open 2505 samples [usecs] 93 2012 1068605 525242687 close 2504 samples [usecs] 64 3090 343577 71579991 mknod 2501 samples [usecs] 170 1966 932856 408674204 unlink 2500 samples [usecs] 263 57240 1714237 6094549799 mkdir 2 samples [usecs] 789 1903 2692 4243930 rmdir 2 samples [usecs] 2484 3024 5508 15314832 rename 2500 samples [usecs] 232 22055 4110143 10612723473 getattr 3760 samples [usecs] 7 2141 106597 13247675 parallel_rename_file 2500 samples [usecs] 232 22055 4110143 10612723473 crossdir_rename 2500 samples [usecs] 232 22055 4110143 10612723473 PASS 81d (32s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 23:01:19 (1772683279) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7555 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7555 PASS 82 (2s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 23:01:21 (1772683281) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (1s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 23:01:23 (1772683283) fail_loc=0x8000060b PASS 84 (7s) == sanityn test 85: Lustre API root cache race =========== 23:01:30 (1772683290) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1772683291 Finishing test test1 at 1772683292 PASS 85 (4s) == sanityn test 90: open/create and unlink striped directory ========================================================== 23:01:34 (1772683294) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (1s) == sanityn test 91: chmod and unlink striped directory === 23:01:35 (1772683295) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (1s) == sanityn test 92: create remote directory under orphan directory ========================================================== 23:01:36 (1772683296) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (2s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 23:01:38 (1772683298) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5861 0x16e5 0x240000400 1 5826 0x16c2 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5827 0x16c3 0x280000400 0 5862 0x16e6 0x240000400 rc1=0 and rc2=0 PASS 93 (9s) == sanityn test 94: signal vs CP callback race =========== 23:01:47 (1772683307) osc.lustre-OST0000-osc-ffff9c180325a000.idle_timeout=0 osc.lustre-OST0000-osc-ffff9c1807d63000.idle_timeout=0 osc.lustre-OST0001-osc-ffff9c180325a000.idle_timeout=0 osc.lustre-OST0001-osc-ffff9c1807d63000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0526438 s, 156 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.09713 s, 1.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6133: 288814 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0128281 s, 319 kB/s osc.lustre-OST0000-osc-ffff9c180325a000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9c1807d63000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c180325a000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c1807d63000.idle_timeout=debug PASS 94 (11s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 23:01:58 (1772683318) llite.lustre-ffff9c180325a000.fast_read=0 llite.lustre-ffff9c1807d63000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0303848 s, 69.0 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff9c180325a000.fast_read=1 llite.lustre-ffff9c1807d63000.fast_read=1 PASS 95a (15s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 23:02:13 (1772683333) llite.lustre-ffff9c180325a000.fast_read=0 llite.lustre-ffff9c1807d63000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.0025995 s, 4.7 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff9c180325a000.fast_read=1 llite.lustre-ffff9c1807d63000.fast_read=1 PASS 95b (8s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 23:02:21 (1772683341) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (1s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 23:02:22 (1772683342) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00206775 s, 2.0 MB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (3s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 23:02:25 (1772683345) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0429672 s, 48.8 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (2s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 23:02:27 (1772683347) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0454241 s, 46.2 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (3s) == sanityn test 100e: DoM: read on open and file size ==== 23:02:30 (1772683350) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00197532 s, 6.1 MB/s PASS 100e (2s) == sanityn test 101a: Discard DoM data on unlink ========= 23:02:32 (1772683352) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00192047 s, 2.1 MB/s PASS 101a (3s) == sanityn test 101b: Discard DoM data on rename ========= 23:02:35 (1772683355) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00160245 s, 2.6 MB/s PASS 101b (2s) == sanityn test 101c: Discard DoM data on close-unlink === 23:02:37 (1772683357) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00247717 s, 1.7 MB/s PASS 101c (4s) SKIP: sanityn test_102 skipping ALWAYS excluded test 102 == sanityn test 103: Test size correctness with lockahead ========================================================== 23:02:42 (1772683362) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1772683363 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1772683365 Finishing test test23 at 1772683366 PASS 103 (7s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 23:02:49 (1772683369) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (2s) == sanityn test 105: Glimpse and lock cancel race ======== 23:02:51 (1772683371) fail_loc=0x416 PASS 105 (23s) == sanityn test 106a: Verify the btime via statx() ======= 23:03:14 (1772683394) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (1s) == sanityn test 106b: Glimpse RPCs test for statx ======== 23:03:15 (1772683395) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00344038 s, 298 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826937738 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-04 23:03:16.000000000 -0500 Modify: 2026-03-04 23:03:16.000000000 -0500 Change: 2026-03-04 23:03:16.000000000 -0500 Birth: 2026-03-04 23:03:16.000000000 -0500 /mnt/lustre/f106b.sanityn 144115238826937738 -rw-r--r-- 1 0 0 1772683396 1772683396 1772683396 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826937738 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-04 23:03:16.000000000 -0500 Modify: 2026-03-04 23:03:16.000000000 -0500 Change: 2026-03-04 23:03:16.000000000 -0500 Birth: 2026-03-04 23:03:16.000000000 -0500 1772683396 1024 1 PASS 106b (3s) == sanityn test 106c: Verify statx attributes mask ======= 23:03:18 (1772683398) PASS 106c (2s) == sanityn test 107a: Basic grouplock conflict =========== 23:03:21 (1772683401) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.110314 s, 95.1 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5870 0x16ee 0x240000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7555 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7555 First grouplock blocks second one PASS 107a (4s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 23:03:25 (1772683405) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.102303 s, 102 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5834 0x16ca 0x280000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7555 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (9s) == sanityn test 108a: lseek: parallel updates ============ 23:03:34 (1772683414) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.0022418 s, 4.5 MB/s llite.lustre-ffff9c180325a000.hybrid_io=0 llite.lustre-ffff9c1807d63000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.25138 s, 2.0 MB/s PASS 108a (6s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 23:03:41 (1772683421) 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre (opts:) 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) Starting race between client mount instances (50 iterations): Iteration 1 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre (opts:) Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 2 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre (opts:) Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 3 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre (opts:) Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 4 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg111-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 5 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre (opts:) Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 6 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg111-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 7 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre (opts:) Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 8 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre (opts:) Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 9 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre (opts:) Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 10 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre (opts:) Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 11 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre (opts:) Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 12 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre (opts:) Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 13 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre (opts:) Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 14 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg111-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 15 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre (opts:) Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 16 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre (opts:) Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 17 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg111-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 18 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre (opts:) Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 19 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre (opts:) Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 20 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg111-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 21 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg111-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 22 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre (opts:) Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 23 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre (opts:) Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 24 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg111-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 25 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg111-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 26 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg111-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 27 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre (opts:) Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 28 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre (opts:) Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 29 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre (opts:) Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 30 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre (opts:) Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 31 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre (opts:) Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 32 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre (opts:) Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 33 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg111-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 34 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre (opts:) Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 35 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg111-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 36 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg111-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 37 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre (opts:) Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 38 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg111-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 39 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre (opts:) Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 40 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg111-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 41 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre (opts:) Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 42 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg111-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 43 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg111-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 44 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre (opts:) Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 45 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre (opts:) Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 46 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre (opts:) Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 47 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre (opts:) Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 48 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre (opts:) Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 49 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg111-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Iteration 50 Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 192.168.201.111@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.111@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg111-client.virtnet /mnt/lustre (opts:) Stopping client oleg111-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre2 PASS 109 (511s) == sanityn test 110: do not grant another lock on resend ========================================================== 23:12:12 (1772683932) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-04 23:12:12.000000000 -0500 Modify: 2026-03-04 23:12:12.000000000 -0500 Change: 2026-03-04 23:12:12.000000000 -0500 Birth: 2026-03-04 23:12:13.000000000 -0500 fail_loc=0 fail_val=0 Connected clients: oleg111-client.virtnet oleg111-client.virtnet PASS 110 (64s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 23:13:16 (1772683996) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (1s) == sanityn test 112: update max-inherit in default LMV === 23:13:17 (1772683997) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (1s) == sanityn test 113: check servers of specified fs ======= 23:13:18 (1772683998) lustre-OST0000-osc-ffff9c18108da800 active. lustre-OST0001-osc-ffff9c18108da800 active. lustre-OST0000-osc-ffff9c18108da800 active. lustre-OST0001-osc-ffff9c18108da800 active. MGC192.168.201.111@tcp active. PASS 113 (2s) == sanityn test 114: implicit default LMV inherit ======== 23:13:20 (1772684000) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (1s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 23:13:21 (1772684001) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (1s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 23:13:22 (1772684002) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (1s) == sanityn test 121: trunc append race =================== 23:13:23 (1772684003) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00249035 s, 8.2 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.011744 s, 1.7 MB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (4s) == sanityn test 200: service remains healthy while able to process request ========================================================== 23:13:27 (1772684007) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0839 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0865 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0884 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0918 s, 0.2 kB/s1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0862 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0999 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0926 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1457 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1386 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1642 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1479 s, 0.1 kB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1464 s, 0.1 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1552 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1634 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2334 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2463 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2304 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2553 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg111-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9c18083de800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c18083de800.ost_server_uuid in IDLE state after 0 sec PASS 200 (66s) cleanup: ====================================================== == sanityn test complete, duration 5688 sec ============== 23:14:34 (1772684074) === sanityn: start cleanup 23:14:34 (1772684074) === Stopping clients: oleg111-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg111-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 23:15:20 (1772684120) ===