== sanity test 182b: Test parallel modify metadata operations from osp ========================================================== 04:18:50 (1788509930)
osp.lustre-MDT0000-osp-MDT0001.rpc_stats
osp.lustre-MDT0001-osp-MDT0000.rpc_stats
total: 10 mkdir in 0.69 seconds: 14.40 ops/second
 - mkdir 135 (time 1788509954.78 total 10.00 last 13.50)
 - mkdir 277 (time 1788509964.87 total 20.09 last 14.08)
 - mkdir 417 (time 1788509974.88 total 30.10 last 13.98)
 - mkdir 562 (time 1788509984.89 total 40.11 last 14.48)
 - mkdir 712 (time 1788509994.89 total 50.11 last 15.00)
 - mkdir 865 (time 1788510004.96 total 60.18 last 15.20)
 - mkdir 995 (time 1788510014.98 total 70.20 last 12.97)
total: 1000 mkdir in 70.58 seconds: 14.17 ops/second
 - mkdir 132 (time 1788509954.13 total 10.03 last 13.15)
 - mkdir 268 (time 1788509964.14 total 20.04 last 13.60)
 - mkdir 403 (time 1788509974.14 total 30.04 last 13.50)
 - mkdir 551 (time 1788509984.19 total 40.09 last 14.72)
 - mkdir 696 (time 1788509994.21 total 50.11 last 14.47)
 - mkdir 844 (time 1788510004.23 total 60.13 last 14.77)
 - mkdir 971 (time 1788510014.26 total 70.16 last 12.67)
total: 1000 mkdir in 73.59 seconds: 13.59 ops/second
 - mkdir 131 (time 1788509954.57 total 10.02 last 13.07)
 - mkdir 269 (time 1788509964.57 total 20.02 last 13.80)
 - mkdir 409 (time 1788509974.58 total 30.03 last 13.99)
 - mkdir 554 (time 1788509984.62 total 40.07 last 14.44)
 - mkdir 695 (time 1788509994.65 total 50.11 last 14.05)
 - mkdir 846 (time 1788510004.67 total 60.12 last 15.08)
 - mkdir 974 (time 1788510014.71 total 70.16 last 12.75)
total: 1000 mkdir in 73.28 seconds: 13.65 ops/second
 - mkdir 131 (time 1788509954.78 total 10.03 last 13.06)
 - mkdir 262 (time 1788509964.89 total 20.14 last 12.96)
 - mkdir 402 (time 1788509974.89 total 30.14 last 14.00)
 - mkdir 549 (time 1788509984.91 total 40.16 last 14.67)
 - mkdir 695 (time 1788509994.99 total 50.24 last 14.49)
 - mkdir 842 (time 1788510005.02 total 60.27 last 14.66)
 - mkdir 972 (time 1788510015.05 total 70.30 last 12.96)
total: 1000 mkdir in 73.74 seconds: 13.56 ops/second
 - mkdir 136 (time 1788509955.17 total 10.04 last 13.55)
 - mkdir 270 (time 1788509965.25 total 20.12 last 13.29)
 - mkdir 413 (time 1788509975.30 total 30.16 last 14.24)
 - mkdir 562 (time 1788509985.38 total 40.25 last 14.77)
 - mkdir 710 (time 1788509995.43 total 50.29 last 14.74)
 - mkdir 854 (time 1788510005.44 total 60.30 last 14.38)
 - mkdir 972 (time 1788510015.50 total 70.36 last 11.73)
total: 1000 mkdir in 74.71 seconds: 13.39 ops/second
 - mkdir 129 (time 1788509954.57 total 10.03 last 12.86)
 - mkdir 266 (time 1788509964.60 total 20.06 last 13.66)
 - mkdir 403 (time 1788509974.61 total 30.08 last 13.68)
 - mkdir 545 (time 1788509984.62 total 40.08 last 14.19)
 - mkdir 687 (time 1788509994.64 total 50.10 last 14.17)
 - mkdir 833 (time 1788510004.68 total 60.15 last 14.54)
 - mkdir 956 (time 1788510014.71 total 70.17 last 12.27)
total: 1000 mkdir in 75.86 seconds: 13.18 ops/second
 - mkdir 135 (time 1788509955.12 total 10.06 last 13.42)
 - mkdir 268 (time 1788509965.25 total 20.19 last 13.13)
 - mkdir 406 (time 1788509975.25 total 30.20 last 13.80)
 - mkdir 555 (time 1788509985.31 total 40.25 last 14.82)
 - mkdir 697 (time 1788509995.31 total 50.25 last 14.20)
 - mkdir 847 (time 1788510005.34 total 60.28 last 14.95)
 - mkdir 969 (time 1788510015.41 total 70.36 last 12.11)
total: 1000 mkdir in 75.35 seconds: 13.27 ops/second
 - mkdir 134 (time 1788509954.93 total 10.02 last 13.38)
 - mkdir 269 (time 1788509964.94 total 20.03 last 13.49)
 - mkdir 408 (time 1788509974.97 total 30.06 last 13.86)
 - mkdir 550 (time 1788509985.00 total 40.09 last 14.15)
 - mkdir 687 (time 1788509995.04 total 50.13 last 13.65)
 - mkdir 838 (time 1788510005.06 total 60.15 last 15.06)
 - mkdir 961 (time 1788510015.10 total 70.19 last 12.25)
total: 1000 mkdir in 75.86 seconds: 13.18 ops/second
 - mkdir 127 (time 1788509954.03 total 10.08 last 12.59)
 - mkdir 261 (time 1788509964.11 total 20.16 last 13.30)
 - mkdir 399 (time 1788509974.14 total 30.19 last 13.77)
 - mkdir 545 (time 1788509984.27 total 40.32 last 14.41)
 - mkdir 685 (time 1788509994.31 total 50.36 last 13.95)
 - mkdir 828 (time 1788510004.32 total 60.37 last 14.28)
 - mkdir 953 (time 1788510014.44 total 70.49 last 12.35)
total: 1000 mkdir in 76.96 seconds: 12.99 ops/second
 - mkdir 126 (time 1788509951.64 total 10.03 last 12.57)
 - mkdir 250 (time 1788509961.65 total 20.04 last 12.39)
 - mkdir 385 (time 1788509971.66 total 30.05 last 13.49)
 - mkdir 522 (time 1788509981.74 total 40.12 last 13.60)
 - mkdir 657 (time 1788509991.80 total 50.19 last 13.41)
 - mkdir 790 (time 1788510001.82 total 60.21 last 13.27)
 - mkdir 914 (time 1788510011.86 total 70.24 last 12.36)
 - mkdir 999 (time 1788510021.95 total 80.33 last 8.42)
total: 1000 mkdir in 80.33 seconds: 12.45 ops/second
Time for file creation 89 sec for 8 parallel RPCs
rmdir(/mnt/lustre/d182b.sanity/5/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/6/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/1/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/3/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/8/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/0/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/9/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/7/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/4/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/2/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
Time for file removal 16 sec for 8 parallel RPCs
osp.lustre-MDT0001-osp-MDT0000.max_mod_rpcs_in_flight=1
total: 10 mkdir in 0.81 seconds: 12.38 ops/second
 - mkdir 104 (time 1788510698.27 total 10.04 last 10.36)
 - mkdir 223 (time 1788510708.31 total 20.07 last 11.86)
 - mkdir 346 (time 1788510718.33 total 30.10 last 12.27)
 - mkdir 473 (time 1788510728.37 total 40.14 last 12.65)
 - mkdir 595 (time 1788510738.42 total 50.19 last 12.14)
 - mkdir 725 (time 1788510748.50 total 60.26 last 12.91)
 - mkdir 850 (time 1788510758.52 total 70.28 last 12.47)
 - mkdir 976 (time 1788510768.56 total 80.32 last 12.55)
total: 1000 mkdir in 82.37 seconds: 12.14 ops/second
 - mkdir 111 (time 1788510700.89 total 10.02 last 11.08)
 - mkdir 225 (time 1788510710.91 total 20.04 last 11.38)
 - mkdir 347 (time 1788510720.92 total 30.05 last 12.18)
 - mkdir 475 (time 1788510731.01 total 40.14 last 12.69)
 - mkdir 605 (time 1788510741.04 total 50.17 last 12.96)
 - mkdir 732 (time 1788510751.05 total 60.18 last 12.68)
 - mkdir 861 (time 1788510761.10 total 70.23 last 12.84)
 - mkdir 978 (time 1788510771.19 total 80.32 last 11.60)
total: 1000 mkdir in 83.38 seconds: 11.99 ops/second
 - mkdir 117 (time 1788510701.20 total 10.03 last 11.66)
 - mkdir 233 (time 1788510711.24 total 20.07 last 11.56)
 - mkdir 354 (time 1788510721.27 total 30.10 last 12.07)
 - mkdir 481 (time 1788510731.37 total 40.20 last 12.57)
 - mkdir 614 (time 1788510741.45 total 50.28 last 13.20)
 - mkdir 737 (time 1788510751.48 total 60.31 last 12.27)
 - mkdir 864 (time 1788510761.49 total 70.31 last 12.69)
 - mkdir 980 (time 1788510771.50 total 80.33 last 11.58)
total: 1000 mkdir in 83.17 seconds: 12.02 ops/second
 - mkdir 113 (time 1788510700.74 total 10.03 last 11.27)
 - mkdir 228 (time 1788510710.80 total 20.09 last 11.43)
 - mkdir 348 (time 1788510720.83 total 30.12 last 11.97)
 - mkdir 476 (time 1788510730.84 total 40.12 last 12.79)
 - mkdir 606 (time 1788510740.84 total 50.12 last 13.00)
 - mkdir 729 (time 1788510750.90 total 60.19 last 12.22)
 - mkdir 853 (time 1788510760.98 total 70.27 last 12.30)
 - mkdir 973 (time 1788510771.03 total 80.32 last 11.94)
total: 1000 mkdir in 84.53 seconds: 11.83 ops/second
 - mkdir 100 (time 1788510698.52 total 10.03 last 9.97)
 - mkdir 211 (time 1788510708.56 total 20.06 last 11.06)
 - mkdir 325 (time 1788510718.62 total 30.13 last 11.32)
 - mkdir 453 (time 1788510728.67 total 40.18 last 12.74)
 - mkdir 576 (time 1788510738.75 total 50.26 last 12.20)
 - mkdir 707 (time 1788510748.75 total 60.26 last 13.10)
 - mkdir 827 (time 1788510758.76 total 70.27 last 11.99)
 - mkdir 946 (time 1788510768.81 total 80.31 last 11.84)
total: 1000 mkdir in 88.09 seconds: 11.35 ops/second
 - mkdir 114 (time 1788510701.17 total 10.06 last 11.33)
 - mkdir 232 (time 1788510711.19 total 20.08 last 11.78)
 - mkdir 350 (time 1788510721.20 total 30.08 last 11.79)
 - mkdir 474 (time 1788510731.22 total 40.11 last 12.37)
 - mkdir 607 (time 1788510741.26 total 50.15 last 13.25)
 - mkdir 731 (time 1788510751.29 total 60.17 last 12.37)
 - mkdir 853 (time 1788510761.37 total 70.25 last 12.10)
 - mkdir 968 (time 1788510771.38 total 80.26 last 11.49)
total: 1000 mkdir in 85.66 seconds: 11.67 ops/second
 - mkdir 105 (time 1788510700.06 total 10.06 last 10.44)
 - mkdir 220 (time 1788510710.09 total 20.09 last 11.46)
 - mkdir 333 (time 1788510720.19 total 30.19 last 11.19)
 - mkdir 456 (time 1788510730.24 total 40.23 last 12.24)
 - mkdir 583 (time 1788510740.28 total 50.28 last 12.64)
 - mkdir 713 (time 1788510750.33 total 60.33 last 12.94)
 - mkdir 832 (time 1788510760.37 total 70.37 last 11.85)
 - mkdir 952 (time 1788510770.40 total 80.40 last 11.97)
total: 1000 mkdir in 87.84 seconds: 11.38 ops/second
 - mkdir 105 (time 1788510700.59 total 10.02 last 10.48)
 - mkdir 214 (time 1788510710.65 total 20.08 last 10.83)
 - mkdir 333 (time 1788510720.70 total 30.13 last 11.84)
 - mkdir 460 (time 1788510730.78 total 40.21 last 12.60)
 - mkdir 588 (time 1788510740.84 total 50.27 last 12.72)
 - mkdir 718 (time 1788510750.86 total 60.29 last 12.98)
 - mkdir 843 (time 1788510760.88 total 70.32 last 12.47)
 - mkdir 959 (time 1788510770.96 total 80.39 last 11.51)
total: 1000 mkdir in 87.29 seconds: 11.46 ops/second
 - mkdir 107 (time 1788510700.89 total 10.02 last 10.68)
 - mkdir 217 (time 1788510710.96 total 20.09 last 10.92)
 - mkdir 337 (time 1788510720.96 total 30.09 last 12.00)
 - mkdir 461 (time 1788510731.07 total 40.20 last 12.26)
 - mkdir 590 (time 1788510741.11 total 50.25 last 12.84)
 - mkdir 716 (time 1788510751.17 total 60.30 last 12.54)
 - mkdir 838 (time 1788510761.18 total 70.32 last 12.18)
 - mkdir 957 (time 1788510771.27 total 80.40 last 11.80)
total: 1000 mkdir in 87.39 seconds: 11.44 ops/second
 - mkdir 100 (time 1788510701.09 total 10.02 last 9.98)
 - mkdir 204 (time 1788510711.17 total 20.10 last 10.32)
 - mkdir 314 (time 1788510721.18 total 30.11 last 10.99)
 - mkdir 434 (time 1788510731.21 total 40.14 last 11.96)
 - mkdir 561 (time 1788510741.23 total 50.16 last 12.68)
 - mkdir 688 (time 1788510751.27 total 60.20 last 12.64)
 - mkdir 803 (time 1788510761.33 total 70.26 last 11.43)
 - mkdir 916 (time 1788510771.45 total 80.38 last 11.16)
 - mkdir 987 (time 1788510781.45 total 90.38 last 7.10)
total: 1000 mkdir in 91.02 seconds: 10.99 ops/second
Time for file creation 105 sec for 1 RPC sent at a time
 - unlinked 0 (time 1788510791 ; total 0 ; last 0)
total: 1000 unlinks in 43 seconds: 23.255814 unlinks/second
 - unlinked 0 (time 1788510793 ; total 0 ; last 0)
total: 1000 unlinks in 44 seconds: 22.727272 unlinks/second
 - unlinked 0 (time 1788510793 ; total 0 ; last 0)
total: 1000 unlinks in 45 seconds: 22.222221 unlinks/second
 - unlinked 0 (time 1788510794 ; total 0 ; last 0)
total: 1000 unlinks in 46 seconds: 21.739130 unlinks/second
 - unlinked 0 (time 1788510794 ; total 0 ; last 0)
total: 1000 unlinks in 46 seconds: 21.739130 unlinks/second
 - unlinked 0 (time 1788510794 ; total 0 ; last 0)
total: 1000 unlinks in 46 seconds: 21.739130 unlinks/second
 - unlinked 0 (time 1788510794 ; total 0 ; last 0)
total: 1000 unlinks in 47 seconds: 21.276596 unlinks/second
 - unlinked 0 (time 1788510794 ; total 0 ; last 0)
total: 1000 unlinks in 47 seconds: 21.276596 unlinks/second
 - unlinked 0 (time 1788510794 ; total 0 ; last 0)
total: 1000 unlinks in 47 seconds: 21.276596 unlinks/second
 - unlinked 0 (time 1788510795 ; total 1 ; last 1)
total: 1000 unlinks in 48 seconds: 20.833334 unlinks/second
Time for file removal 61 sec for 1 RPC sent at a time
osp.lustre-MDT0001-osp-MDT0000.max_mod_rpcs_in_flight=8