== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:02:56 (1788973376) start mds service on oleg420-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 9 02:32 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg420-server' oleg420-server: oleg420-server.virtnet: executing load_modules_local oleg420-server: kmemleak disabled oleg420-server: Loading modules from /home/green/git/lustre-release/lustre oleg420-server: detected 4 online CPUs by sysfs oleg420-server: MODOPTS_LIBCFS= oleg420-server: Force libcfs to create 2 CPU partitions oleg420-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg420-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg420-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg420-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg420-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0000 oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 176 (time 1788973472.56 total 10.04 last 17.54) - open/close 345 (time 1788973482.98 total 20.45 last 16.23) - open/close 495 (time 1788973492.98 total 30.46 last 14.99) - open/close 588 (time 1788973503.01 total 40.48 last 9.27) - open/close 694 (time 1788973513.03 total 50.50 last 10.58) - open/close 781 (time 1788973523.31 total 60.78 last 8.47) - open/close 905 (time 1788973533.33 total 70.80 last 12.37) - open/close 1125 (time 1788973544.80 total 82.27 last 19.19) - open/close 1240 (time 1788973555.63 total 93.10 last 10.62) - open/close 1432 (time 1788973566.69 total 104.16 last 17.36) - open/close 1578 (time 1788973576.99 total 114.46 last 14.17) - open/close 1706 (time 1788973587.19 total 124.66 last 12.54) - open/close 1980 (time 1788973598.18 total 135.66 last 24.92) - open/close 2145 (time 1788973609.00 total 146.47 last 15.26) - open/close 2258 (time 1788973619.03 total 156.50 last 11.26) - open/close 2387 (time 1788973629.06 total 166.53 last 12.87) - open/close 2629 (time 1788973639.20 total 176.67 last 23.86) - open/close 2812 (time 1788973649.57 total 187.04 last 17.64) - open/close 2940 (time 1788973660.56 total 198.04 last 11.64) - open/close 3042 (time 1788973670.58 total 208.05 last 10.18) - open/close 3163 (time 1788973681.26 total 218.73 last 11.34) - open/close 3398 (time 1788973691.26 total 228.73 last 23.48) - open/close 3581 (time 1788973702.17 total 239.65 last 16.77) - open/close 3760 (time 1788973712.21 total 249.68 last 17.83) - open/close 4015 (time 1788973722.23 total 259.70 last 25.47) - open/close 4215 (time 1788973732.71 total 270.18 last 19.08) - open/close 4449 (time 1788973743.34 total 280.81 last 22.00) - open/close 4710 (time 1788973753.35 total 290.82 last 26.08) - open/close 4869 (time 1788973764.08 total 301.55 last 14.82) - open/close 5074 (time 1788973774.49 total 311.96 last 19.69) - open/close 5315 (time 1788973784.49 total 321.96 last 24.10) - open/close 5533 (time 1788973794.97 total 332.44 last 20.81) - open/close 5819 (time 1788973804.98 total 342.45 last 28.57) - open/close 5962 (time 1788973821.16 total 358.63 last 8.84) - open/close 6211 (time 1788973831.78 total 369.25 last 23.45) - open/close 6370 (time 1788973841.80 total 379.27 last 15.87) - open/close 6524 (time 1788973852.01 total 389.48 last 15.08) - open/close 6691 (time 1788973862.43 total 399.90 last 16.03) - open/close 6796 (time 1788973872.44 total 409.91 last 10.49) - open/close 6964 (time 1788973882.45 total 419.92 last 16.78) - open/close 7209 (time 1788973892.57 total 430.04 last 24.19) - open/close 7390 (time 1788973902.58 total 440.05 last 18.09) - open/close 7554 (time 1788973913.56 total 451.03 last 14.93) - open/close 7647 (time 1788973923.57 total 461.04 last 9.29) - open/close 7830 (time 1788973934.06 total 471.53 last 17.45) - open/close 7976 (time 1788973944.81 total 482.28 last 13.59) - open/close 8119 (time 1788973955.57 total 493.04 last 13.29) - open/close 8258 (time 1788973965.92 total 503.39 last 13.43) - open/close 8387 (time 1788973975.93 total 513.40 last 12.89) - open/close 8544 (time 1788973986.13 total 523.61 last 15.38) - open/close 8659 (time 1788973996.15 total 533.62 last 11.48) - open/close 8788 (time 1788974006.17 total 543.64 last 12.88) - open/close 9030 (time 1788974016.18 total 553.65 last 24.17) - open/close 9210 (time 1788974026.97 total 564.44 last 16.69) - open/close 9448 (time 1788974037.16 total 574.64 last 23.34) - open/close 9608 (time 1788974048.85 total 586.32 last 13.70) - open/close 9825 (time 1788974059.80 total 597.27 last 19.82) - open/close 10000 (time 1788974069.67 total 607.14 last 17.72) - open/close 10163 (time 1788974080.47 total 617.94 last 15.09) - open/close 10317 (time 1788974090.47 total 627.94 last 15.40) - open/close 10479 (time 1788974101.34 total 638.81 last 14.90) - open/close 10699 (time 1788974111.96 total 649.43 last 20.72) - open/close 10936 (time 1788974122.34 total 659.81 last 22.83) - open/close 11135 (time 1788974134.07 total 671.54 last 16.96) - open/close 11258 (time 1788974144.08 total 681.55 last 12.29) - open/close 11339 (time 1788974154.15 total 691.62 last 8.04) - open/close 11603 (time 1788974165.16 total 702.63 last 23.97) - open/close 11781 (time 1788974175.19 total 712.66 last 17.76) - open/close 11960 (time 1788974185.19 total 722.66 last 17.90) - open/close 12076 (time 1788974195.49 total 732.96 last 11.26) - open/close 12319 (time 1788974205.50 total 742.97 last 24.27) - open/close 12499 (time 1788974216.16 total 753.63 last 16.89) - open/close 12691 (time 1788974227.05 total 764.52 last 17.63) - open/close 12916 (time 1788974237.86 total 775.33 last 20.81) - open/close 13120 (time 1788974247.92 total 785.39 last 20.28) - open/close 13316 (time 1788974258.13 total 795.61 last 19.19) - open/close 13562 (time 1788974268.14 total 805.62 last 24.58) - open/close 13751 (time 1788974278.76 total 816.23 last 17.80) - open/close 13880 (time 1788974289.91 total 827.38 last 11.57) - open/close 13978 (time 1788974300.44 total 837.91 last 9.31) - open/close 14147 (time 1788974310.61 total 848.08 last 16.62) - open/close 14281 (time 1788974321.15 total 858.62 last 12.71) - open/close 14477 (time 1788974335.71 total 873.18 last 13.46) - open/close 14639 (time 1788974346.34 total 883.81 last 15.24) - open/close 14838 (time 1788974356.35 total 893.82 last 19.89) - open/close 14978 (time 1788974367.39 total 904.86 last 12.67) - open/close 15111 (time 1788974377.84 total 915.32 last 12.73) - open/close 15309 (time 1788974387.85 total 925.32 last 19.78) - open/close 15454 (time 1788974398.47 total 935.94 last 13.66) - open/close 15648 (time 1788974408.47 total 945.94 last 19.39) - open/close 15829 (time 1788974419.24 total 956.72 last 16.80) - open/close 16095 (time 1788974429.34 total 966.81 last 26.36) - open/close 16326 (time 1788974439.34 total 976.82 last 23.08) - open/close 16420 (time 1788974449.86 total 987.33 last 8.94) - open/close 16634 (time 1788974460.90 total 998.37 last 19.38) - open/close 16817 (time 1788974470.90 total 1008.38 last 18.29) - open/close 17086 (time 1788974481.75 total 1019.23 last 24.79) - open/close 17248 (time 1788974491.77 total 1029.24 last 16.18) - open/close 17553 (time 1788974501.78 total 1039.25 last 30.47) - open/close 18181 (time 1788974511.79 total 1049.26 last 62.72) - open/close 18857 (time 1788974521.79 total 1059.26 last 67.59) - open/close 19437 (time 1788974531.80 total 1069.27 last 57.93) - open/close 20000 (time 1788974540.98 total 1078.45 last 61.36) - open/close 20676 (time 1788974550.99 total 1088.46 last 67.52) - open/close 21212 (time 1788974561.00 total 1098.47 last 53.56) - open/close 21805 (time 1788974571.01 total 1108.48 last 59.21) - open/close 22230 (time 1788974581.02 total 1118.50 last 42.44) - open/close 22745 (time 1788974591.04 total 1128.51 last 51.41) - open/close 23289 (time 1788974601.07 total 1138.54 last 54.26) - open/close 23836 (time 1788974611.07 total 1148.54 last 54.70) - open/close 24430 (time 1788974621.07 total 1158.54 last 59.39) - open/close 24980 (time 1788974631.07 total 1168.54 last 55.00) - open/close 25583 (time 1788974641.08 total 1178.55 last 60.25) - open/close 26126 (time 1788974651.09 total 1188.56 last 54.25) - open/close 26670 (time 1788974661.10 total 1198.57 last 54.35) - open/close 27284 (time 1788974671.11 total 1208.58 last 61.32) - open/close 27890 (time 1788974681.12 total 1218.59 last 60.52) - open/close 28492 (time 1788974691.16 total 1228.64 last 59.96) - open/close 29108 (time 1788974701.17 total 1238.65 last 61.53) - open/close 29717 (time 1788974711.18 total 1248.65 last 60.86) - open/close 30000 (time 1788974715.74 total 1253.21 last 62.13) - open/close 30600 (time 1788974725.75 total 1263.22 last 59.94) - open/close 31230 (time 1788974735.75 total 1273.22 last 62.99) - open/close 31780 (time 1788974745.75 total 1283.22 last 54.99) - open/close 32408 (time 1788974755.75 total 1293.22 last 62.78) - open/close 32952 (time 1788974765.76 total 1303.24 last 54.34) - open/close 33524 (time 1788974775.77 total 1313.24 last 57.18) - open/close 34099 (time 1788974785.78 total 1323.25 last 57.45) - open/close 34713 (time 1788974795.78 total 1333.25 last 61.36) - open/close 35373 (time 1788974805.81 total 1343.28 last 65.84) - open/close 36041 (time 1788974815.81 total 1353.28 last 66.79) - open/close 36606 (time 1788974825.82 total 1363.29 last 56.46) - open/close 37213 (time 1788974835.83 total 1373.30 last 60.63) - open/close 37731 (time 1788974845.83 total 1383.31 last 51.76) - open/close 38323 (time 1788974855.83 total 1393.31 last 59.20) - open/close 38918 (time 1788974865.85 total 1403.32 last 59.41) - open/close 39541 (time 1788974875.85 total 1413.33 last 62.26) - open/close 40000 (time 1788974884.46 total 1421.93 last 53.34) - open/close 40634 (time 1788974894.47 total 1431.95 last 63.30) - open/close 41216 (time 1788974904.49 total 1441.96 last 58.14) - open/close 41786 (time 1788974914.49 total 1451.96 last 56.99) - open/close 42385 (time 1788974924.49 total 1461.97 last 59.86) - open/close 43069 (time 1788974934.49 total 1471.97 last 68.39) - open/close 43720 (time 1788974944.50 total 1481.97 last 65.05) - open/close 44317 (time 1788974954.51 total 1491.98 last 59.68) - open/close 44903 (time 1788974964.52 total 1501.99 last 58.50) - open/close 45467 (time 1788974974.53 total 1512.00 last 56.35) - open/close 46108 (time 1788974984.54 total 1522.01 last 64.07) - open/close 46711 (time 1788974994.56 total 1532.03 last 60.17) - open/close 47275 (time 1788975004.56 total 1542.03 last 56.37) - open/close 47808 (time 1788975014.57 total 1552.04 last 53.28) - open/close 48349 (time 1788975024.58 total 1562.05 last 54.02) - open/close 48929 (time 1788975034.59 total 1572.06 last 57.97) - open/close 49488 (time 1788975044.59 total 1582.06 last 55.90) total: 50000 open/close in 1591.41 seconds: 31.42 ops/second - unlinked 0 (time 1788975059 ; total 0 ; last 0) - unlinked 10000 (time 1788975233 ; total 174 ; last 174) - unlinked 20000 (time 1788975412 ; total 353 ; last 179) - unlinked 30000 (time 1788975597 ; total 538 ; last 185) - unlinked 40000 (time 1788975770 ; total 711 ; last 173) total: 50000 unlinks in 882 seconds: 56.689342 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg420-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg420-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg420-server stop mds service on oleg420-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg420-server unloading modules via unload_modules_local on: 'oleg420-server' oleg420-server: oleg420-server.virtnet: executing unload_modules_local oleg420-server: kmemleak disabled modules unloaded. pdsh@oleg420-client: oleg420-client: ssh exited with exit code 2 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 2 pdsh@oleg420-client: oleg420-client: ssh exited with exit code 2 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 2