== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:14:36 (1788747276) start mds service on oleg428-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 6 20:21 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg428-server' oleg428-server: oleg428-server.virtnet: executing load_modules_local oleg428-server: kmemleak disabled oleg428-server: Loading modules from /home/green/git/lustre-release/lustre oleg428-server: detected 4 online CPUs by sysfs oleg428-server: MODOPTS_LIBCFS= oleg428-server: Force libcfs to create 2 CPU partitions oleg428-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg428-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg428-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg428-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg428-server: oleg428-server.virtnet: executing set_default_debug -1 all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg428-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg428-server: oleg428-server.virtnet: executing set_default_debug -1 all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-OST0000 oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg428-client.virtnet: -o user_xattr,flock 192.168.204.128@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 228 (time 1788747360.39 total 10.02 last 22.76) - open/close 416 (time 1788747371.44 total 21.07 last 17.01) - open/close 658 (time 1788747381.45 total 31.08 last 24.18) - open/close 901 (time 1788747391.46 total 41.09 last 24.28) - open/close 1060 (time 1788747401.47 total 51.10 last 15.88) - open/close 1297 (time 1788747412.28 total 61.91 last 21.92) - open/close 1541 (time 1788747422.65 total 72.27 last 23.54) - open/close 1737 (time 1788747433.02 total 82.65 last 18.89) - open/close 1914 (time 1788747443.03 total 92.66 last 17.69) - open/close 1969 (time 1788747453.50 total 103.13 last 5.25) - open/close 2116 (time 1788747464.13 total 113.76 last 13.82) - open/close 2364 (time 1788747474.16 total 123.79 last 24.73) - open/close 2508 (time 1788747484.43 total 134.06 last 14.02) - open/close 2764 (time 1788747495.29 total 144.92 last 23.58) - open/close 3013 (time 1788747506.37 total 155.99 last 22.48) - open/close 3265 (time 1788747517.37 total 166.99 last 22.91) - open/close 3493 (time 1788747529.10 total 178.73 last 19.42) - open/close 3663 (time 1788747539.11 total 188.74 last 16.99) - open/close 3824 (time 1788747549.78 total 199.40 last 15.10) - open/close 4053 (time 1788747559.79 total 209.42 last 22.87) - open/close 4208 (time 1788747570.17 total 219.79 last 14.94) - open/close 4444 (time 1788747580.29 total 229.92 last 23.32) - open/close 4680 (time 1788747590.30 total 239.92 last 23.58) - open/close 4854 (time 1788747600.75 total 250.38 last 16.64) - open/close 5053 (time 1788747611.51 total 261.14 last 18.50) - open/close 5198 (time 1788747621.70 total 271.32 last 14.24) - open/close 5436 (time 1788747634.51 total 284.14 last 18.57) - open/close 5581 (time 1788747644.92 total 294.55 last 13.93) - open/close 5714 (time 1788747655.29 total 304.92 last 12.82) - open/close 5945 (time 1788747665.29 total 314.92 last 23.10) - open/close 6087 (time 1788747675.30 total 324.92 last 14.20) - open/close 6257 (time 1788747685.32 total 334.95 last 16.95) - open/close 6412 (time 1788747695.86 total 345.49 last 14.71) - open/close 6536 (time 1788747705.88 total 355.51 last 12.37) - open/close 6727 (time 1788747715.93 total 365.56 last 19.01) - open/close 6882 (time 1788747726.39 total 376.02 last 14.81) - open/close 7033 (time 1788747737.30 total 386.93 last 13.85) - open/close 7189 (time 1788747747.93 total 397.56 last 14.68) - open/close 7274 (time 1788747758.55 total 408.17 last 8.01) - open/close 7485 (time 1788747768.56 total 418.19 last 21.07) - open/close 7641 (time 1788747778.58 total 428.21 last 15.57) - open/close 7749 (time 1788747788.59 total 438.21 last 10.79) - open/close 7871 (time 1788747799.02 total 448.65 last 11.69) - open/close 8066 (time 1788747809.15 total 458.78 last 19.25) - open/close 8185 (time 1788747819.53 total 469.16 last 11.47) - open/close 8374 (time 1788747829.55 total 479.18 last 18.86) - open/close 8540 (time 1788747839.99 total 489.62 last 15.90) - open/close 8754 (time 1788747850.00 total 499.62 last 21.38) - open/close 8859 (time 1788747860.01 total 509.63 last 10.49) - open/close 9087 (time 1788747870.01 total 519.63 last 22.80) - open/close 9295 (time 1788747880.02 total 529.65 last 20.78) - open/close 9527 (time 1788747890.70 total 540.33 last 21.72) - open/close 9717 (time 1788747901.62 total 551.24 last 17.40) - open/close 9943 (time 1788747911.62 total 561.25 last 22.59) - open/close 10000 (time 1788747918.18 total 567.80 last 8.70) - open/close 10199 (time 1788747929.06 total 578.68 last 18.29) - open/close 10389 (time 1788747939.24 total 588.87 last 18.66) - open/close 10522 (time 1788747950.42 total 600.05 last 11.89) - open/close 10729 (time 1788747960.46 total 610.09 last 20.62) - open/close 10899 (time 1788747970.47 total 620.09 last 16.99) - open/close 11017 (time 1788747981.27 total 630.90 last 10.92) - open/close 11149 (time 1788747991.82 total 641.44 last 12.52) - open/close 11338 (time 1788748001.82 total 651.45 last 18.88) - open/close 11421 (time 1788748012.23 total 661.85 last 7.98) - open/close 11676 (time 1788748022.24 total 671.86 last 25.47) - open/close 11756 (time 1788748032.24 total 681.87 last 7.99) - open/close 11873 (time 1788748042.51 total 692.13 last 11.40) - open/close 11991 (time 1788748052.74 total 702.37 last 11.53) - open/close 12082 (time 1788748063.81 total 713.44 last 8.22) - open/close 12203 (time 1788748073.82 total 723.45 last 12.09) - open/close 12379 (time 1788748083.84 total 733.47 last 17.56) - open/close 12560 (time 1788748093.87 total 743.50 last 18.05) - open/close 12807 (time 1788748104.70 total 754.32 last 22.81) - open/close 13065 (time 1788748114.70 total 764.32 last 25.80) - open/close 13275 (time 1788748124.92 total 774.55 last 20.54) - open/close 13492 (time 1788748135.42 total 785.05 last 20.66) - open/close 13755 (time 1788748146.43 total 796.06 last 23.90) - open/close 14002 (time 1788748156.62 total 806.25 last 24.23) - open/close 14176 (time 1788748166.73 total 816.35 last 17.22) - open/close 14256 (time 1788748177.56 total 827.19 last 7.38) - open/close 14340 (time 1788748188.48 total 838.11 last 7.69) - open/close 14527 (time 1788748198.49 total 848.11 last 18.69) - open/close 14679 (time 1788748208.51 total 858.14 last 15.17) - open/close 14767 (time 1788748220.17 total 869.80 last 7.55) - open/close 14926 (time 1788748230.95 total 880.57 last 14.75) - open/close 15093 (time 1788748241.42 total 891.05 last 15.95) - open/close 15270 (time 1788748251.78 total 901.41 last 17.08) - open/close 15518 (time 1788748262.76 total 912.39 last 22.58) - open/close 15720 (time 1788748273.02 total 922.64 last 19.71) - open/close 15982 (time 1788748283.45 total 933.08 last 25.10) - open/close 16195 (time 1788748293.96 total 943.58 last 20.28) - open/close 16388 (time 1788748304.29 total 953.92 last 18.68) - open/close 16556 (time 1788748315.13 total 964.76 last 15.50) - open/close 16771 (time 1788748325.39 total 975.01 last 20.96) - open/close 16925 (time 1788748335.40 total 985.02 last 15.39) - open/close 17116 (time 1788748345.55 total 995.17 last 18.82) - open/close 17292 (time 1788748355.55 total 1005.18 last 17.59) - open/close 17623 (time 1788748365.56 total 1015.18 last 33.08) - open/close 18287 (time 1788748375.57 total 1025.20 last 66.32) - open/close 18941 (time 1788748385.59 total 1035.21 last 65.30) - open/close 19571 (time 1788748395.62 total 1045.25 last 62.77) - open/close 20000 (time 1788748402.32 total 1051.94 last 64.06) - open/close 20623 (time 1788748412.33 total 1061.96 last 62.22) - open/close 21270 (time 1788748422.34 total 1071.97 last 64.65) - open/close 21936 (time 1788748432.36 total 1081.98 last 66.48) - open/close 22509 (time 1788748442.36 total 1091.98 last 57.30) - open/close 23082 (time 1788748452.37 total 1101.99 last 57.25) - open/close 23662 (time 1788748462.38 total 1112.00 last 57.94) - open/close 24237 (time 1788748472.42 total 1122.04 last 57.27) - open/close 24882 (time 1788748482.42 total 1132.04 last 64.50) - open/close 25518 (time 1788748492.42 total 1142.05 last 63.58) - open/close 26129 (time 1788748502.43 total 1152.05 last 61.06) - open/close 26715 (time 1788748512.45 total 1162.07 last 58.50) - open/close 27335 (time 1788748522.45 total 1172.08 last 61.95) - open/close 27985 (time 1788748532.47 total 1182.09 last 64.92) - open/close 28605 (time 1788748542.48 total 1192.11 last 61.89) - open/close 29231 (time 1788748552.51 total 1202.14 last 62.43) - open/close 29873 (time 1788748562.52 total 1212.14 last 64.17) - open/close 30000 (time 1788748564.56 total 1214.19 last 62.17) - open/close 30701 (time 1788748574.57 total 1224.19 last 70.04) - open/close 31307 (time 1788748584.58 total 1234.20 last 60.54) - open/close 31891 (time 1788748594.58 total 1244.21 last 58.37) - open/close 32486 (time 1788748604.59 total 1254.22 last 59.45) - open/close 33119 (time 1788748614.59 total 1264.22 last 63.30) - open/close 33712 (time 1788748624.60 total 1274.23 last 59.25) - open/close 34340 (time 1788748634.60 total 1284.23 last 62.77) - open/close 35016 (time 1788748644.62 total 1294.24 last 67.52) - open/close 35699 (time 1788748654.65 total 1304.28 last 68.06) - open/close 36375 (time 1788748664.66 total 1314.28 last 67.55) - open/close 37063 (time 1788748674.66 total 1324.29 last 68.76) - open/close 37677 (time 1788748684.68 total 1334.30 last 61.33) - open/close 38284 (time 1788748694.68 total 1344.30 last 60.69) - open/close 38905 (time 1788748704.69 total 1354.31 last 62.05) - open/close 39528 (time 1788748714.69 total 1364.31 last 62.30) - open/close 40000 (time 1788748722.36 total 1371.98 last 61.53) - open/close 40676 (time 1788748732.36 total 1381.99 last 67.57) - open/close 41367 (time 1788748742.36 total 1391.99 last 69.10) - open/close 41987 (time 1788748752.38 total 1402.01 last 61.88) - open/close 42679 (time 1788748762.39 total 1412.02 last 69.14) - open/close 43313 (time 1788748772.43 total 1422.05 last 63.18) - open/close 43994 (time 1788748782.44 total 1432.07 last 68.01) - open/close 44631 (time 1788748792.44 total 1442.07 last 63.69) - open/close 45261 (time 1788748802.45 total 1452.08 last 62.94) - open/close 45948 (time 1788748812.45 total 1462.08 last 68.68) - open/close 46631 (time 1788748822.46 total 1472.09 last 68.22) - open/close 47264 (time 1788748832.48 total 1482.10 last 63.23) - open/close 47897 (time 1788748842.50 total 1492.12 last 63.18) - open/close 48503 (time 1788748852.50 total 1502.12 last 60.60) - open/close 49202 (time 1788748862.50 total 1512.13 last 69.85) - open/close 49872 (time 1788748872.51 total 1522.14 last 66.95) total: 50000 open/close in 1523.99 seconds: 32.81 ops/second - unlinked 0 (time 1788748877 ; total 0 ; last 0) - unlinked 10000 (time 1788749029 ; total 152 ; last 152) - unlinked 20000 (time 1788749190 ; total 313 ; last 161) - unlinked 30000 (time 1788749333 ; total 456 ; last 143) - unlinked 40000 (time 1788749472 ; total 595 ; last 139) total: 50000 unlinks in 762 seconds: 65.616798 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg428-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg428-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg428-server stop mds service on oleg428-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg428-server unloading modules via unload_modules_local on: 'oleg428-server' oleg428-server: oleg428-server.virtnet: executing unload_modules_local oleg428-server: kmemleak disabled modules unloaded. pdsh@oleg428-client: oleg428-client: ssh exited with exit code 2 pdsh@oleg428-client: oleg428-server: ssh exited with exit code 2 pdsh@oleg428-client: oleg428-client: ssh exited with exit code 2 pdsh@oleg428-client: oleg428-server: ssh exited with exit code 2