== sanity test 182b: Test parallel modify metadata operations from osp ========================================================== 03:10:41 (1781248241)
osp.lustre-MDT0000-osp-MDT0001.rpc_stats
osp.lustre-MDT0001-osp-MDT0000.rpc_stats
total: 10 mkdir in 0.59 seconds: 16.99 ops/second
 - mkdir 126 (time 1781248261.88 total 10.02 last 12.58)
 - mkdir 230 (time 1781248271.91 total 20.05 last 10.37)
 - mkdir 349 (time 1781248281.97 total 30.11 last 11.83)
 - mkdir 468 (time 1781248291.99 total 40.13 last 11.88)
 - mkdir 586 (time 1781248302.20 total 50.34 last 11.56)
 - mkdir 678 (time 1781248312.28 total 60.42 last 9.13)
 - mkdir 763 (time 1781248322.33 total 70.47 last 8.46)
 - mkdir 884 (time 1781248332.43 total 80.57 last 11.98)
total: 1000 mkdir in 89.90 seconds: 11.12 ops/second
 - mkdir 144 (time 1781248261.28 total 10.12 last 14.23)
 - mkdir 242 (time 1781248271.37 total 20.21 last 9.71)
 - mkdir 358 (time 1781248281.38 total 30.22 last 11.59)
 - mkdir 470 (time 1781248291.43 total 40.27 last 11.14)
 - mkdir 581 (time 1781248301.49 total 50.33 last 11.03)
 - mkdir 673 (time 1781248311.61 total 60.45 last 9.09)
 - mkdir 758 (time 1781248321.65 total 70.49 last 8.47)
 - mkdir 876 (time 1781248331.65 total 80.49 last 11.79)
 - mkdir 994 (time 1781248341.72 total 90.56 last 11.72)
total: 1000 mkdir in 91.00 seconds: 10.99 ops/second
 - mkdir 130 (time 1781248261.87 total 10.14 last 12.81)
 - mkdir 230 (time 1781248271.93 total 20.20 last 9.94)
 - mkdir 351 (time 1781248281.96 total 30.23 last 12.06)
 - mkdir 460 (time 1781248291.97 total 40.25 last 10.88)
 - mkdir 572 (time 1781248302.10 total 50.37 last 11.06)
 - mkdir 660 (time 1781248312.21 total 60.49 last 8.70)
 - mkdir 741 (time 1781248322.27 total 70.54 last 8.06)
 - mkdir 863 (time 1781248332.29 total 80.57 last 12.17)
 - mkdir 981 (time 1781248342.40 total 90.67 last 11.68)
total: 1000 mkdir in 92.61 seconds: 10.80 ops/second
 - mkdir 128 (time 1781248261.56 total 10.06 last 12.73)
 - mkdir 224 (time 1781248271.56 total 20.06 last 9.60)
 - mkdir 337 (time 1781248281.65 total 30.16 last 11.19)
 - mkdir 447 (time 1781248291.81 total 40.31 last 10.83)
 - mkdir 562 (time 1781248301.87 total 50.38 last 11.43)
 - mkdir 652 (time 1781248311.90 total 60.40 last 8.98)
 - mkdir 736 (time 1781248321.97 total 70.47 last 8.34)
 - mkdir 856 (time 1781248332.01 total 80.51 last 11.96)
 - mkdir 972 (time 1781248342.05 total 90.55 last 11.55)
total: 1000 mkdir in 93.59 seconds: 10.69 ops/second
 - mkdir 126 (time 1781248262.05 total 10.08 last 12.50)
 - mkdir 230 (time 1781248272.10 total 20.14 last 10.34)
 - mkdir 347 (time 1781248282.19 total 30.22 last 11.60)
 - mkdir 460 (time 1781248292.20 total 40.23 last 11.29)
 - mkdir 566 (time 1781248302.27 total 50.31 last 10.52)
 - mkdir 657 (time 1781248312.32 total 60.35 last 9.06)
 - mkdir 737 (time 1781248322.36 total 70.40 last 7.96)
 - mkdir 861 (time 1781248332.40 total 80.44 last 12.35)
 - mkdir 975 (time 1781248342.43 total 90.47 last 11.37)
total: 1000 mkdir in 93.12 seconds: 10.74 ops/second
 - mkdir 123 (time 1781248261.88 total 10.06 last 12.23)
 - mkdir 222 (time 1781248271.92 total 20.11 last 9.86)
 - mkdir 339 (time 1781248281.96 total 30.14 last 11.66)
 - mkdir 449 (time 1781248292.06 total 40.25 last 10.88)
 - mkdir 559 (time 1781248302.20 total 50.39 last 10.85)
 - mkdir 644 (time 1781248312.27 total 60.45 last 8.44)
 - mkdir 727 (time 1781248322.27 total 70.46 last 8.30)
 - mkdir 848 (time 1781248332.31 total 80.49 last 12.06)
 - mkdir 964 (time 1781248342.37 total 90.55 last 11.53)
total: 1000 mkdir in 94.51 seconds: 10.58 ops/second
 - mkdir 125 (time 1781248262.40 total 10.15 last 12.32)
 - mkdir 228 (time 1781248272.44 total 20.18 last 10.26)
 - mkdir 343 (time 1781248282.44 total 30.19 last 11.49)
 - mkdir 459 (time 1781248292.63 total 40.38 last 11.38)
 - mkdir 566 (time 1781248302.79 total 50.54 last 10.53)
 - mkdir 655 (time 1781248312.84 total 60.59 last 8.85)
 - mkdir 739 (time 1781248322.90 total 70.65 last 8.35)
 - mkdir 854 (time 1781248332.99 total 80.74 last 11.39)
 - mkdir 968 (time 1781248343.02 total 90.77 last 11.36)
total: 1000 mkdir in 94.31 seconds: 10.60 ops/second
 - mkdir 129 (time 1781248261.96 total 10.25 last 12.58)
 - mkdir 225 (time 1781248272.00 total 20.29 last 9.56)
 - mkdir 339 (time 1781248282.07 total 30.36 last 11.33)
 - mkdir 447 (time 1781248292.07 total 40.36 last 10.80)
 - mkdir 551 (time 1781248302.20 total 50.49 last 10.27)
 - mkdir 636 (time 1781248312.27 total 60.56 last 8.44)
 - mkdir 716 (time 1781248322.35 total 70.64 last 7.93)
 - mkdir 835 (time 1781248332.42 total 80.71 last 11.82)
 - mkdir 946 (time 1781248342.47 total 90.76 last 11.05)
total: 1000 mkdir in 96.66 seconds: 10.35 ops/second
 - mkdir 117 (time 1781248262.24 total 10.13 last 11.56)
 - mkdir 216 (time 1781248272.26 total 20.15 last 9.88)
 - mkdir 325 (time 1781248282.33 total 30.22 last 10.82)
 - mkdir 436 (time 1781248292.36 total 40.24 last 11.07)
 - mkdir 542 (time 1781248302.47 total 50.35 last 10.48)
 - mkdir 625 (time 1781248312.51 total 60.39 last 8.27)
 - mkdir 710 (time 1781248322.59 total 70.47 last 8.43)
 - mkdir 825 (time 1781248332.61 total 80.49 last 11.48)
 - mkdir 938 (time 1781248342.66 total 90.54 last 11.25)
total: 1000 mkdir in 96.81 seconds: 10.33 ops/second
 - mkdir 129 (time 1781248261.21 total 10.12 last 12.75)
 - mkdir 224 (time 1781248271.30 total 20.21 last 9.41)
 - mkdir 333 (time 1781248281.33 total 30.24 last 10.86)
 - mkdir 438 (time 1781248291.43 total 40.34 last 10.40)
 - mkdir 544 (time 1781248301.49 total 50.40 last 10.54)
 - mkdir 623 (time 1781248311.53 total 60.45 last 7.86)
 - mkdir 701 (time 1781248321.69 total 70.61 last 7.68)
 - mkdir 813 (time 1781248331.71 total 80.63 last 11.18)
 - mkdir 925 (time 1781248341.72 total 90.63 last 11.20)
total: 1000 mkdir in 98.24 seconds: 10.18 ops/second
Time for file creation 106 sec for 8 parallel RPCs
rmdir(/mnt/lustre/d182b.sanity/0/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/7/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/4/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/2/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/8/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/9/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/5/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/3/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/1/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/6/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
Time for file removal 13 sec for 8 parallel RPCs
osp.lustre-MDT0001-osp-MDT0000.max_mod_rpcs_in_flight=1
total: 10 mkdir in 0.60 seconds: 16.65 ops/second
 - mkdir 134 (time 1781248964.55 total 10.04 last 13.34)
 - mkdir 269 (time 1781248974.58 total 20.07 last 13.46)
 - mkdir 417 (time 1781248984.58 total 30.07 last 14.80)
 - mkdir 575 (time 1781248994.60 total 40.09 last 15.77)
 - mkdir 728 (time 1781249004.67 total 50.16 last 15.19)
 - mkdir 879 (time 1781249014.73 total 60.22 last 15.02)
total: 1000 mkdir in 68.36 seconds: 14.63 ops/second
 - mkdir 123 (time 1781248962.46 total 10.01 last 12.29)
 - mkdir 262 (time 1781248972.52 total 20.07 last 13.82)
 - mkdir 395 (time 1781248982.54 total 30.09 last 13.28)
 - mkdir 546 (time 1781248992.56 total 40.11 last 15.06)
 - mkdir 703 (time 1781249002.67 total 50.22 last 15.53)
 - mkdir 841 (time 1781249012.68 total 60.22 last 13.79)
 - mkdir 980 (time 1781249022.68 total 70.23 last 13.90)
total: 1000 mkdir in 72.58 seconds: 13.78 ops/second
 - mkdir 128 (time 1781248963.45 total 10.09 last 12.69)
 - mkdir 268 (time 1781248973.57 total 20.21 last 13.84)
 - mkdir 404 (time 1781248983.57 total 30.21 last 13.60)
 - mkdir 558 (time 1781248993.57 total 40.21 last 15.40)
 - mkdir 712 (time 1781249003.71 total 50.35 last 15.19)
 - mkdir 853 (time 1781249013.72 total 60.35 last 14.09)
 - mkdir 989 (time 1781249023.73 total 70.36 last 13.59)
total: 1000 mkdir in 71.83 seconds: 13.92 ops/second
 - mkdir 133 (time 1781248964.82 total 10.07 last 13.21)
 - mkdir 266 (time 1781248974.88 total 20.13 last 13.22)
 - mkdir 410 (time 1781248984.89 total 30.14 last 14.39)
 - mkdir 564 (time 1781248994.90 total 40.15 last 15.38)
 - mkdir 711 (time 1781249004.90 total 50.15 last 14.70)
 - mkdir 855 (time 1781249014.92 total 60.17 last 14.37)
 - mkdir 989 (time 1781249025.01 total 70.26 last 13.29)
total: 1000 mkdir in 71.49 seconds: 13.99 ops/second
 - mkdir 129 (time 1781248964.38 total 10.02 last 12.87)
 - mkdir 264 (time 1781248974.40 total 20.04 last 13.48)
 - mkdir 408 (time 1781248984.43 total 30.07 last 14.35)
 - mkdir 562 (time 1781248994.45 total 40.09 last 15.38)
 - mkdir 707 (time 1781249004.55 total 50.19 last 14.36)
 - mkdir 856 (time 1781249014.59 total 60.23 last 14.83)
 - mkdir 984 (time 1781249024.65 total 70.29 last 12.73)
total: 1000 mkdir in 71.96 seconds: 13.90 ops/second
 - mkdir 126 (time 1781248963.87 total 10.09 last 12.49)
 - mkdir 263 (time 1781248973.97 total 20.19 last 13.56)
 - mkdir 401 (time 1781248983.97 total 30.19 last 13.80)
 - mkdir 557 (time 1781248993.99 total 40.21 last 15.57)
 - mkdir 702 (time 1781249004.02 total 50.24 last 14.46)
 - mkdir 846 (time 1781249014.07 total 60.29 last 14.32)
 - mkdir 975 (time 1781249024.12 total 70.33 last 12.84)
total: 1000 mkdir in 73.87 seconds: 13.54 ops/second
 - mkdir 123 (time 1781248964.08 total 10.08 last 12.21)
 - mkdir 256 (time 1781248974.10 total 20.10 last 13.27)
 - mkdir 392 (time 1781248984.10 total 30.10 last 13.60)
 - mkdir 546 (time 1781248994.15 total 40.15 last 15.33)
 - mkdir 697 (time 1781249004.21 total 50.21 last 15.01)
 - mkdir 843 (time 1781249014.26 total 60.26 last 14.52)
 - mkdir 976 (time 1781249024.42 total 70.42 last 13.09)
total: 1000 mkdir in 73.75 seconds: 13.56 ops/second
 - mkdir 133 (time 1781248964.90 total 10.00 last 13.29)
 - mkdir 267 (time 1781248974.96 total 20.07 last 13.31)
 - mkdir 412 (time 1781248984.99 total 30.10 last 14.46)
 - mkdir 561 (time 1781248995.04 total 40.15 last 14.82)
 - mkdir 704 (time 1781249005.09 total 50.20 last 14.23)
 - mkdir 850 (time 1781249015.11 total 60.22 last 14.57)
 - mkdir 977 (time 1781249025.17 total 70.28 last 12.62)
total: 1000 mkdir in 73.24 seconds: 13.65 ops/second
 - mkdir 128 (time 1781248964.64 total 10.00 last 12.80)
 - mkdir 259 (time 1781248974.69 total 20.05 last 13.04)
 - mkdir 394 (time 1781248984.73 total 30.09 last 13.45)
 - mkdir 550 (time 1781248994.81 total 40.16 last 15.48)
 - mkdir 692 (time 1781249004.83 total 50.19 last 14.17)
 - mkdir 838 (time 1781249014.86 total 60.21 last 14.56)
 - mkdir 965 (time 1781249024.96 total 70.32 last 12.57)
total: 1000 mkdir in 74.88 seconds: 13.35 ops/second
 - mkdir 121 (time 1781248964.15 total 10.06 last 12.03)
 - mkdir 253 (time 1781248974.20 total 20.10 last 13.14)
 - mkdir 385 (time 1781248984.20 total 30.11 last 13.19)
 - mkdir 535 (time 1781248994.25 total 40.15 last 14.93)
 - mkdir 675 (time 1781249004.30 total 50.21 last 13.92)
 - mkdir 810 (time 1781249014.30 total 60.21 last 13.50)
 - mkdir 936 (time 1781249024.36 total 70.27 last 12.52)
total: 1000 mkdir in 77.15 seconds: 12.96 ops/second
Time for file creation 88 sec for 1 RPC sent at a time
 - unlinked 0 (time 1781249040 ; total 0 ; last 0)
total: 1000 unlinks in 38 seconds: 26.315790 unlinks/second
 - unlinked 0 (time 1781249041 ; total 1 ; last 1)
total: 1000 unlinks in 38 seconds: 26.315790 unlinks/second
 - unlinked 0 (time 1781249040 ; total 0 ; last 0)
total: 1000 unlinks in 39 seconds: 25.641026 unlinks/second
 - unlinked 0 (time 1781249042 ; total 0 ; last 0)
total: 1000 unlinks in 37 seconds: 27.027027 unlinks/second
 - unlinked 0 (time 1781249042 ; total 0 ; last 0)
total: 1000 unlinks in 38 seconds: 26.315790 unlinks/second
 - unlinked 0 (time 1781249042 ; total 0 ; last 0)
total: 1000 unlinks in 39 seconds: 25.641026 unlinks/second
 - unlinked 0 (time 1781249042 ; total 0 ; last 0)
total: 1000 unlinks in 39 seconds: 25.641026 unlinks/second
 - unlinked 0 (time 1781249043 ; total 0 ; last 0)
total: 1000 unlinks in 38 seconds: 26.315790 unlinks/second
 - unlinked 0 (time 1781249043 ; total 0 ; last 0)
total: 1000 unlinks in 38 seconds: 26.315790 unlinks/second
 - unlinked 0 (time 1781249042 ; total 0 ; last 0)
total: 1000 unlinks in 39 seconds: 25.641026 unlinks/second
Time for file removal 53 sec for 1 RPC sent at a time
osp.lustre-MDT0001-osp-MDT0000.max_mod_rpcs_in_flight=8