== sanity test 182b: Test parallel modify metadata operations from osp ========================================================== 01:03:12 (1788670992)
osp.lustre-MDT0000-osp-MDT0001.rpc_stats
osp.lustre-MDT0001-osp-MDT0000.rpc_stats
total: 10 mkdir in 0.46 seconds: 21.70 ops/second
 - mkdir 161 (time 1788671013.16 total 10.01 last 16.08)
 - mkdir 326 (time 1788671023.18 total 20.03 last 16.47)
 - mkdir 493 (time 1788671033.18 total 30.03 last 16.69)
 - mkdir 638 (time 1788671043.24 total 40.09 last 14.42)
 - mkdir 790 (time 1788671053.26 total 50.11 last 15.16)
 - mkdir 933 (time 1788671063.28 total 60.13 last 14.28)
total: 1000 mkdir in 64.08 seconds: 15.61 ops/second
 - mkdir 158 (time 1788671012.34 total 10.01 last 15.78)
 - mkdir 318 (time 1788671022.42 total 20.09 last 15.87)
 - mkdir 484 (time 1788671032.44 total 30.11 last 16.57)
 - mkdir 625 (time 1788671042.46 total 40.14 last 14.06)
 - mkdir 766 (time 1788671052.50 total 50.18 last 14.05)
 - mkdir 909 (time 1788671062.59 total 60.26 last 14.18)
total: 1000 mkdir in 66.14 seconds: 15.12 ops/second
 - mkdir 166 (time 1788671014.07 total 10.01 last 16.59)
 - mkdir 326 (time 1788671024.09 total 20.03 last 15.97)
 - mkdir 489 (time 1788671034.15 total 30.09 last 16.20)
 - mkdir 634 (time 1788671044.15 total 40.09 last 14.50)
 - mkdir 783 (time 1788671054.22 total 50.16 last 14.79)
 - mkdir 922 (time 1788671064.25 total 60.19 last 13.86)
total: 1000 mkdir in 66.20 seconds: 15.11 ops/second
 - mkdir 167 (time 1788671013.65 total 10.02 last 16.66)
 - mkdir 322 (time 1788671023.66 total 20.04 last 15.48)
 - mkdir 486 (time 1788671033.78 total 30.16 last 16.21)
 - mkdir 629 (time 1788671043.79 total 40.16 last 14.29)
 - mkdir 771 (time 1788671053.86 total 50.24 last 14.09)
 - mkdir 913 (time 1788671063.87 total 60.25 last 14.19)
total: 1000 mkdir in 66.71 seconds: 14.99 ops/second
 - mkdir 162 (time 1788671013.08 total 10.09 last 16.05)
 - mkdir 318 (time 1788671023.09 total 20.11 last 15.58)
 - mkdir 483 (time 1788671033.20 total 30.21 last 16.33)
 - mkdir 625 (time 1788671043.24 total 40.25 last 14.15)
 - mkdir 766 (time 1788671053.29 total 50.31 last 14.02)
 - mkdir 897 (time 1788671063.30 total 60.31 last 13.10)
total: 1000 mkdir in 67.92 seconds: 14.72 ops/second
 - mkdir 159 (time 1788671013.53 total 10.05 last 15.82)
 - mkdir 316 (time 1788671023.64 total 20.16 last 15.53)
 - mkdir 479 (time 1788671033.64 total 30.16 last 16.29)
 - mkdir 618 (time 1788671043.68 total 40.20 last 13.85)
 - mkdir 766 (time 1788671053.70 total 50.22 last 14.77)
 - mkdir 901 (time 1788671063.75 total 60.27 last 13.44)
total: 1000 mkdir in 67.77 seconds: 14.76 ops/second
 - mkdir 164 (time 1788671013.84 total 10.03 last 16.35)
 - mkdir 315 (time 1788671023.86 total 20.04 last 15.08)
 - mkdir 482 (time 1788671033.93 total 30.12 last 16.57)
 - mkdir 625 (time 1788671043.94 total 40.12 last 14.29)
 - mkdir 764 (time 1788671053.94 total 50.13 last 13.90)
 - mkdir 901 (time 1788671063.99 total 60.18 last 13.63)
total: 1000 mkdir in 68.02 seconds: 14.70 ops/second
 - mkdir 156 (time 1788671013.59 total 10.05 last 15.52)
 - mkdir 312 (time 1788671023.61 total 20.06 last 15.58)
 - mkdir 480 (time 1788671033.64 total 30.10 last 16.74)
 - mkdir 621 (time 1788671043.65 total 40.10 last 14.09)
 - mkdir 759 (time 1788671053.65 total 50.10 last 13.80)
 - mkdir 898 (time 1788671063.71 total 60.17 last 13.81)
total: 1000 mkdir in 68.92 seconds: 14.51 ops/second
 - mkdir 161 (time 1788671014.24 total 10.01 last 16.08)
 - mkdir 319 (time 1788671024.29 total 20.06 last 15.73)
 - mkdir 488 (time 1788671034.35 total 30.12 last 16.80)
 - mkdir 629 (time 1788671044.37 total 40.14 last 14.07)
 - mkdir 767 (time 1788671054.41 total 50.18 last 13.74)
 - mkdir 905 (time 1788671064.46 total 60.23 last 13.73)
total: 1000 mkdir in 68.47 seconds: 14.60 ops/second
 - mkdir 151 (time 1788671014.18 total 10.03 last 15.06)
 - mkdir 302 (time 1788671024.23 total 20.08 last 15.02)
 - mkdir 458 (time 1788671034.26 total 30.11 last 15.55)
 - mkdir 594 (time 1788671044.27 total 40.12 last 13.58)
 - mkdir 727 (time 1788671054.29 total 50.13 last 13.28)
 - mkdir 859 (time 1788671064.34 total 60.19 last 13.13)
 - mkdir 985 (time 1788671074.37 total 70.22 last 12.56)
total: 1000 mkdir in 70.87 seconds: 14.11 ops/second
Time for file creation 81 sec for 8 parallel RPCs
rmdir(/mnt/lustre/d182b.sanity/0/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/2/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/3/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/8/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/5/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/4/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/1/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/9/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/7/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/6/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
Time for file removal 13 sec for 8 parallel RPCs
osp.lustre-MDT0001-osp-MDT0000.max_mod_rpcs_in_flight=1
total: 10 mkdir in 0.65 seconds: 15.43 ops/second
 - mkdir 154 (time 1788671649.83 total 10.04 last 15.34)
 - mkdir 320 (time 1788671659.84 total 20.05 last 16.59)
 - mkdir 446 (time 1788671669.90 total 30.11 last 12.52)
 - mkdir 581 (time 1788671679.97 total 40.18 last 13.40)
 - mkdir 719 (time 1788671689.98 total 50.19 last 13.79)
 - mkdir 898 (time 1788671700.03 total 60.24 last 17.82)
total: 1000 mkdir in 66.27 seconds: 15.09 ops/second
 - mkdir 143 (time 1788671651.69 total 10.01 last 14.29)
 - mkdir 315 (time 1788671661.71 total 20.03 last 17.16)
 - mkdir 439 (time 1788671671.76 total 30.07 last 12.35)
 - mkdir 576 (time 1788671681.81 total 40.13 last 13.63)
 - mkdir 720 (time 1788671691.83 total 50.15 last 14.37)
 - mkdir 911 (time 1788671701.87 total 60.19 last 19.02)
total: 1000 mkdir in 66.71 seconds: 14.99 ops/second
 - mkdir 153 (time 1788671651.90 total 10.03 last 15.25)
 - mkdir 321 (time 1788671661.92 total 20.06 last 16.76)
 - mkdir 444 (time 1788671671.94 total 30.07 last 12.28)
 - mkdir 581 (time 1788671681.96 total 40.10 last 13.67)
 - mkdir 730 (time 1788671691.97 total 50.10 last 14.89)
 - mkdir 912 (time 1788671702.01 total 60.15 last 18.12)
total: 1000 mkdir in 66.93 seconds: 14.94 ops/second
 - mkdir 145 (time 1788671651.09 total 10.07 last 14.41)
 - mkdir 316 (time 1788671661.17 total 20.15 last 16.96)
 - mkdir 438 (time 1788671671.27 total 30.25 last 12.08)
 - mkdir 573 (time 1788671681.28 total 40.25 last 13.50)
 - mkdir 713 (time 1788671691.31 total 50.28 last 13.95)
 - mkdir 899 (time 1788671701.32 total 60.30 last 18.58)
total: 1000 mkdir in 67.95 seconds: 14.72 ops/second
 - mkdir 146 (time 1788671651.43 total 10.01 last 14.58)
 - mkdir 313 (time 1788671661.48 total 20.06 last 16.61)
 - mkdir 434 (time 1788671671.51 total 30.10 last 12.06)
 - mkdir 571 (time 1788671681.59 total 40.18 last 13.59)
 - mkdir 717 (time 1788671691.63 total 50.21 last 14.55)
 - mkdir 897 (time 1788671701.63 total 60.22 last 17.99)
total: 1000 mkdir in 69.00 seconds: 14.49 ops/second
 - mkdir 144 (time 1788671651.49 total 10.00 last 14.40)
 - mkdir 314 (time 1788671661.52 total 20.03 last 16.95)
 - mkdir 438 (time 1788671671.56 total 30.07 last 12.34)
 - mkdir 569 (time 1788671681.65 total 40.16 last 12.99)
 - mkdir 712 (time 1788671691.69 total 50.20 last 14.24)
 - mkdir 893 (time 1788671701.69 total 60.21 last 18.08)
total: 1000 mkdir in 69.71 seconds: 14.35 ops/second
 - mkdir 143 (time 1788671651.75 total 10.06 last 14.22)
 - mkdir 311 (time 1788671661.80 total 20.11 last 16.71)
 - mkdir 435 (time 1788671671.84 total 30.15 last 12.35)
 - mkdir 571 (time 1788671681.85 total 40.16 last 13.60)
 - mkdir 715 (time 1788671691.85 total 50.16 last 14.39)
 - mkdir 896 (time 1788671701.86 total 60.17 last 18.09)
total: 1000 mkdir in 69.62 seconds: 14.36 ops/second
 - mkdir 141 (time 1788671651.69 total 10.05 last 14.04)
 - mkdir 310 (time 1788671661.70 total 20.06 last 16.88)
 - mkdir 430 (time 1788671671.73 total 30.09 last 11.96)
 - mkdir 561 (time 1788671681.76 total 40.11 last 13.07)
 - mkdir 706 (time 1788671691.81 total 50.17 last 14.42)
 - mkdir 888 (time 1788671701.83 total 60.19 last 18.16)
 - mkdir 993 (time 1788671711.88 total 70.24 last 10.45)
total: 1000 mkdir in 70.83 seconds: 14.12 ops/second
 - mkdir 143 (time 1788671651.77 total 10.04 last 14.25)
 - mkdir 310 (time 1788671661.83 total 20.09 last 16.60)
 - mkdir 427 (time 1788671671.86 total 30.13 last 11.66)
 - mkdir 561 (time 1788671681.89 total 40.15 last 13.36)
 - mkdir 704 (time 1788671691.91 total 50.17 last 14.28)
 - mkdir 891 (time 1788671701.93 total 60.19 last 18.67)
 - mkdir 993 (time 1788671712.03 total 70.30 last 10.09)
total: 1000 mkdir in 70.87 seconds: 14.11 ops/second
 - mkdir 139 (time 1788671650.99 total 10.02 last 13.87)
 - mkdir 296 (time 1788671661.03 total 20.07 last 15.63)
 - mkdir 413 (time 1788671671.04 total 30.08 last 11.69)
 - mkdir 542 (time 1788671681.05 total 40.09 last 12.89)
 - mkdir 686 (time 1788671691.07 total 50.11 last 14.37)
 - mkdir 876 (time 1788671701.07 total 60.11 last 18.99)
 - mkdir 983 (time 1788671711.09 total 70.12 last 10.69)
total: 1000 mkdir in 72.00 seconds: 13.89 ops/second
Time for file creation 84 sec for 1 RPC sent at a time
 - unlinked 0 (time 1788671722 ; total 0 ; last 0)
total: 1000 unlinks in 34 seconds: 29.411764 unlinks/second
 - unlinked 0 (time 1788671723 ; total 0 ; last 0)
total: 1000 unlinks in 34 seconds: 29.411764 unlinks/second
 - unlinked 0 (time 1788671724 ; total 0 ; last 0)
total: 1000 unlinks in 33 seconds: 30.303030 unlinks/second
 - unlinked 0 (time 1788671723 ; total 0 ; last 0)
total: 1000 unlinks in 34 seconds: 29.411764 unlinks/second
 - unlinked 0 (time 1788671724 ; total 0 ; last 0)
total: 1000 unlinks in 33 seconds: 30.303030 unlinks/second
 - unlinked 0 (time 1788671723 ; total 0 ; last 0)
total: 1000 unlinks in 35 seconds: 28.571428 unlinks/second
 - unlinked 0 (time 1788671723 ; total 0 ; last 0)
total: 1000 unlinks in 35 seconds: 28.571428 unlinks/second
 - unlinked 0 (time 1788671724 ; total 0 ; last 0)
total: 1000 unlinks in 34 seconds: 29.411764 unlinks/second
 - unlinked 0 (time 1788671724 ; total 0 ; last 0)
total: 1000 unlinks in 34 seconds: 29.411764 unlinks/second
 - unlinked 0 (time 1788671724 ; total 0 ; last 0)
total: 1000 unlinks in 35 seconds: 28.571428 unlinks/second
Time for file removal 47 sec for 1 RPC sent at a time
osp.lustre-MDT0001-osp-MDT0000.max_mod_rpcs_in_flight=8