== sanity test 182b: Test parallel modify metadata operations from osp ========================================================== 01:10:29 (1781241029)
osp.lustre-MDT0000-osp-MDT0001.rpc_stats
osp.lustre-MDT0001-osp-MDT0000.rpc_stats
total: 10 mkdir in 0.76 seconds: 13.08 ops/second
 - mkdir 125 (time 1781241053.47 total 10.01 last 12.49)
 - mkdir 239 (time 1781241063.54 total 20.07 last 11.33)
 - mkdir 346 (time 1781241073.62 total 30.16 last 10.61)
 - mkdir 471 (time 1781241083.72 total 40.25 last 12.38)
 - mkdir 587 (time 1781241093.78 total 50.32 last 11.53)
 - mkdir 698 (time 1781241103.83 total 60.37 last 11.04)
 - mkdir 805 (time 1781241113.84 total 70.38 last 10.69)
 - mkdir 923 (time 1781241123.87 total 80.40 last 11.77)
total: 1000 mkdir in 85.73 seconds: 11.66 ops/second
 - mkdir 118 (time 1781241052.66 total 10.11 last 11.68)
 - mkdir 236 (time 1781241062.73 total 20.18 last 11.72)
 - mkdir 334 (time 1781241072.82 total 30.27 last 9.71)
 - mkdir 450 (time 1781241082.90 total 40.35 last 11.51)
 - mkdir 566 (time 1781241092.93 total 50.38 last 11.56)
 - mkdir 674 (time 1781241102.96 total 60.41 last 10.77)
 - mkdir 778 (time 1781241112.97 total 70.42 last 10.38)
 - mkdir 885 (time 1781241122.99 total 80.44 last 10.68)
 - mkdir 998 (time 1781241133.17 total 90.62 last 11.10)
total: 1000 mkdir in 90.87 seconds: 11.00 ops/second
 - mkdir 116 (time 1781241052.72 total 10.02 last 11.58)
 - mkdir 231 (time 1781241062.73 total 20.03 last 11.49)
 - mkdir 332 (time 1781241072.80 total 30.10 last 10.02)
 - mkdir 451 (time 1781241082.87 total 40.17 last 11.82)
 - mkdir 565 (time 1781241092.88 total 50.18 last 11.40)
 - mkdir 672 (time 1781241102.91 total 60.20 last 10.67)
 - mkdir 778 (time 1781241112.98 total 70.28 last 10.52)
 - mkdir 887 (time 1781241123.00 total 80.30 last 10.87)
 - mkdir 996 (time 1781241133.19 total 90.49 last 10.70)
total: 1000 mkdir in 90.99 seconds: 10.99 ops/second
 - mkdir 116 (time 1781241053.62 total 10.13 last 11.45)
 - mkdir 232 (time 1781241063.66 total 20.18 last 11.54)
 - mkdir 334 (time 1781241073.72 total 30.23 last 10.14)
 - mkdir 456 (time 1781241083.84 total 40.35 last 12.06)
 - mkdir 569 (time 1781241093.84 total 50.35 last 11.30)
 - mkdir 672 (time 1781241103.88 total 60.39 last 10.26)
 - mkdir 778 (time 1781241113.93 total 70.44 last 10.55)
 - mkdir 891 (time 1781241123.97 total 80.48 last 11.25)
 - mkdir 995 (time 1781241134.11 total 90.62 last 10.26)
total: 1000 mkdir in 91.17 seconds: 10.97 ops/second
 - mkdir 116 (time 1781241053.89 total 10.11 last 11.47)
 - mkdir 231 (time 1781241063.91 total 20.13 last 11.49)
 - mkdir 338 (time 1781241073.91 total 30.13 last 10.69)
 - mkdir 460 (time 1781241083.97 total 40.19 last 12.13)
 - mkdir 567 (time 1781241093.99 total 50.21 last 10.67)
 - mkdir 672 (time 1781241104.03 total 60.25 last 10.46)
 - mkdir 775 (time 1781241114.07 total 70.29 last 10.26)
 - mkdir 888 (time 1781241124.13 total 80.35 last 11.23)
 - mkdir 992 (time 1781241134.34 total 90.56 last 10.19)
total: 1000 mkdir in 91.42 seconds: 10.94 ops/second
 - mkdir 120 (time 1781241054.24 total 10.00 last 11.99)
 - mkdir 233 (time 1781241064.26 total 20.03 last 11.27)
 - mkdir 336 (time 1781241074.29 total 30.06 last 10.27)
 - mkdir 453 (time 1781241084.32 total 40.09 last 11.66)
 - mkdir 559 (time 1781241094.37 total 50.14 last 10.55)
 - mkdir 666 (time 1781241104.53 total 60.30 last 10.53)
 - mkdir 777 (time 1781241114.60 total 70.37 last 11.02)
 - mkdir 894 (time 1781241124.64 total 80.41 last 11.65)
 - mkdir 992 (time 1781241134.89 total 90.65 last 9.57)
total: 1000 mkdir in 91.75 seconds: 10.90 ops/second
 - mkdir 115 (time 1781241053.62 total 10.01 last 11.49)
 - mkdir 229 (time 1781241063.65 total 20.05 last 11.36)
 - mkdir 331 (time 1781241073.70 total 30.09 last 10.16)
 - mkdir 450 (time 1781241083.73 total 40.13 last 11.86)
 - mkdir 558 (time 1781241093.76 total 50.15 last 10.77)
 - mkdir 659 (time 1781241103.79 total 60.19 last 10.07)
 - mkdir 761 (time 1781241113.84 total 70.24 last 10.15)
 - mkdir 875 (time 1781241123.89 total 80.28 last 11.35)
 - mkdir 980 (time 1781241133.90 total 90.30 last 10.48)
total: 1000 mkdir in 93.48 seconds: 10.70 ops/second
 - mkdir 115 (time 1781241054.16 total 10.04 last 11.46)
 - mkdir 221 (time 1781241064.26 total 20.13 last 10.50)
 - mkdir 323 (time 1781241074.29 total 30.16 last 10.17)
 - mkdir 444 (time 1781241084.32 total 40.20 last 12.06)
 - mkdir 542 (time 1781241094.33 total 50.21 last 9.79)
 - mkdir 647 (time 1781241104.36 total 60.23 last 10.48)
 - mkdir 751 (time 1781241114.41 total 70.28 last 10.35)
 - mkdir 864 (time 1781241124.47 total 80.34 last 11.23)
 - mkdir 966 (time 1781241134.50 total 90.37 last 10.17)
total: 1000 mkdir in 94.84 seconds: 10.54 ops/second
 - mkdir 105 (time 1781241051.23 total 10.04 last 10.46)
 - mkdir 216 (time 1781241061.31 total 20.11 last 11.01)
 - mkdir 315 (time 1781241071.35 total 30.16 last 9.86)
 - mkdir 427 (time 1781241081.42 total 40.23 last 11.12)
 - mkdir 529 (time 1781241091.50 total 50.31 last 10.12)
 - mkdir 627 (time 1781241101.52 total 60.33 last 9.77)
 - mkdir 725 (time 1781241111.57 total 70.38 last 9.75)
 - mkdir 829 (time 1781241121.59 total 80.40 last 10.38)
 - mkdir 941 (time 1781241131.70 total 90.50 last 11.08)
total: 1000 mkdir in 98.97 seconds: 10.10 ops/second
 - mkdir 103 (time 1781241051.36 total 10.04 last 10.25)
 - mkdir 209 (time 1781241061.44 total 20.13 last 10.51)
 - mkdir 308 (time 1781241071.67 total 30.36 last 9.68)
 - mkdir 416 (time 1781241081.76 total 40.45 last 10.70)
 - mkdir 520 (time 1781241091.80 total 50.49 last 10.36)
 - mkdir 617 (time 1781241101.86 total 60.55 last 9.64)
 - mkdir 712 (time 1781241111.94 total 70.63 last 9.43)
 - mkdir 815 (time 1781241121.97 total 80.66 last 10.26)
 - mkdir 922 (time 1781241132.20 total 90.89 last 10.46)
total: 1000 mkdir in 100.17 seconds: 9.98 ops/second
Time for file creation 111 sec for 8 parallel RPCs
rmdir(/mnt/lustre/d182b.sanity/6/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/2/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/0/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/5/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/8/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/7/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/3/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/4/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/9/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/1/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
Time for file removal 17 sec for 8 parallel RPCs
osp.lustre-MDT0001-osp-MDT0000.max_mod_rpcs_in_flight=1
total: 10 mkdir in 0.46 seconds: 21.94 ops/second
 - mkdir 180 (time 1781241773.79 total 10.03 last 17.95)
 - mkdir 350 (time 1781241783.80 total 20.04 last 16.98)
 - mkdir 511 (time 1781241793.82 total 30.05 last 16.08)
 - mkdir 677 (time 1781241803.87 total 40.10 last 16.52)
 - mkdir 825 (time 1781241813.89 total 50.13 last 14.76)
 - mkdir 937 (time 1781241823.90 total 60.14 last 11.19)
total: 1000 mkdir in 64.87 seconds: 15.42 ops/second
 - mkdir 173 (time 1781241775.09 total 10.03 last 17.26)
 - mkdir 340 (time 1781241785.13 total 20.06 last 16.65)
 - mkdir 505 (time 1781241795.14 total 30.07 last 16.47)
 - mkdir 667 (time 1781241805.20 total 40.13 last 16.11)
 - mkdir 811 (time 1781241815.21 total 50.14 last 14.39)
 - mkdir 926 (time 1781241825.24 total 60.18 last 11.46)
total: 1000 mkdir in 66.95 seconds: 14.94 ops/second
 - mkdir 171 (time 1781241775.74 total 10.02 last 17.06)
 - mkdir 334 (time 1781241785.76 total 20.04 last 16.27)
 - mkdir 504 (time 1781241795.85 total 30.13 last 16.85)
 - mkdir 670 (time 1781241805.85 total 40.14 last 16.59)
 - mkdir 813 (time 1781241815.87 total 50.16 last 14.28)
 - mkdir 929 (time 1781241825.88 total 60.17 last 11.59)
total: 1000 mkdir in 67.04 seconds: 14.92 ops/second
 - mkdir 167 (time 1781241775.35 total 10.02 last 16.67)
 - mkdir 336 (time 1781241785.42 total 20.09 last 16.78)
 - mkdir 500 (time 1781241795.43 total 30.10 last 16.38)
 - mkdir 660 (time 1781241805.46 total 40.13 last 15.95)
 - mkdir 805 (time 1781241815.59 total 50.26 last 14.31)
 - mkdir 921 (time 1781241825.69 total 60.36 last 11.48)
total: 1000 mkdir in 67.70 seconds: 14.77 ops/second
 - mkdir 170 (time 1781241775.76 total 10.01 last 16.99)
 - mkdir 336 (time 1781241785.77 total 20.02 last 16.58)
 - mkdir 503 (time 1781241795.83 total 30.07 last 16.61)
 - mkdir 664 (time 1781241805.85 total 40.10 last 16.06)
 - mkdir 807 (time 1781241815.87 total 50.12 last 14.27)
 - mkdir 921 (time 1781241825.91 total 60.15 last 11.36)
total: 1000 mkdir in 67.69 seconds: 14.77 ops/second
 - mkdir 171 (time 1781241775.82 total 10.04 last 17.04)
 - mkdir 338 (time 1781241785.82 total 20.04 last 16.70)
 - mkdir 502 (time 1781241795.83 total 30.04 last 16.39)
 - mkdir 664 (time 1781241805.85 total 40.07 last 16.17)
 - mkdir 803 (time 1781241815.89 total 50.11 last 13.84)
 - mkdir 918 (time 1781241825.90 total 60.12 last 11.49)
total: 1000 mkdir in 67.89 seconds: 14.73 ops/second
 - mkdir 169 (time 1781241775.77 total 10.02 last 16.87)
 - mkdir 337 (time 1781241785.81 total 20.06 last 16.73)
 - mkdir 500 (time 1781241795.85 total 30.09 last 16.24)
 - mkdir 662 (time 1781241805.89 total 40.14 last 16.13)
 - mkdir 799 (time 1781241816.03 total 50.28 last 13.51)
 - mkdir 915 (time 1781241826.03 total 60.28 last 11.60)
total: 1000 mkdir in 68.20 seconds: 14.66 ops/second
 - mkdir 169 (time 1781241776.08 total 10.02 last 16.87)
 - mkdir 336 (time 1781241786.19 total 20.13 last 16.51)
 - mkdir 499 (time 1781241796.26 total 30.20 last 16.19)
 - mkdir 663 (time 1781241806.29 total 40.23 last 16.35)
 - mkdir 797 (time 1781241816.32 total 50.25 last 13.37)
 - mkdir 916 (time 1781241826.34 total 60.28 last 11.87)
total: 1000 mkdir in 68.40 seconds: 14.62 ops/second
 - mkdir 166 (time 1781241776.01 total 10.01 last 16.59)
 - mkdir 334 (time 1781241786.05 total 20.05 last 16.73)
 - mkdir 497 (time 1781241796.11 total 30.11 last 16.20)
 - mkdir 660 (time 1781241806.13 total 40.13 last 16.27)
 - mkdir 799 (time 1781241816.23 total 50.23 last 13.76)
 - mkdir 912 (time 1781241826.24 total 60.24 last 11.29)
total: 1000 mkdir in 68.70 seconds: 14.56 ops/second
 - mkdir 167 (time 1781241775.87 total 10.02 last 16.67)
 - mkdir 333 (time 1781241785.93 total 20.08 last 16.51)
 - mkdir 495 (time 1781241795.99 total 30.15 last 16.09)
 - mkdir 659 (time 1781241806.00 total 40.15 last 16.39)
 - mkdir 796 (time 1781241816.01 total 50.16 last 13.69)
 - mkdir 912 (time 1781241826.02 total 60.17 last 11.59)
total: 1000 mkdir in 68.85 seconds: 14.52 ops/second
Time for file creation 79 sec for 1 RPC sent at a time
 - unlinked 0 (time 1781241843 ; total 0 ; last 0)
total: 1000 unlinks in 31 seconds: 32.258064 unlinks/second
 - unlinked 0 (time 1781241843 ; total 0 ; last 0)
total: 1000 unlinks in 33 seconds: 30.303030 unlinks/second
 - unlinked 0 (time 1781241844 ; total 0 ; last 0)
total: 1000 unlinks in 32 seconds: 31.250000 unlinks/second
 - unlinked 0 (time 1781241844 ; total 0 ; last 0)
total: 1000 unlinks in 32 seconds: 31.250000 unlinks/second
 - unlinked 0 (time 1781241844 ; total 0 ; last 0)
total: 1000 unlinks in 33 seconds: 30.303030 unlinks/second
 - unlinked 0 (time 1781241844 ; total 0 ; last 0)
total: 1000 unlinks in 33 seconds: 30.303030 unlinks/second
 - unlinked 0 (time 1781241844 ; total 0 ; last 0)
total: 1000 unlinks in 33 seconds: 30.303030 unlinks/second
 - unlinked 0 (time 1781241844 ; total 0 ; last 0)
total: 1000 unlinks in 33 seconds: 30.303030 unlinks/second
 - unlinked 0 (time 1781241844 ; total 0 ; last 0)
total: 1000 unlinks in 33 seconds: 30.303030 unlinks/second
 - unlinked 0 (time 1781241844 ; total 0 ; last 0)
total: 1000 unlinks in 34 seconds: 29.411764 unlinks/second
Time for file removal 44 sec for 1 RPC sent at a time
osp.lustre-MDT0001-osp-MDT0000.max_mod_rpcs_in_flight=8