== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 17:40:05 (1788471605) start mds service on oleg622-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921072 Sep 3 15:45 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg622-server' oleg622-server: oleg622-server.virtnet: executing load_modules_local oleg622-server: kmemleak disabled oleg622-server: Loading modules from /home/green/git/lustre-release/lustre oleg622-server: detected 4 online CPUs by sysfs oleg622-server: MODOPTS_LIBCFS= oleg622-server: Force libcfs to create 2 CPU partitions oleg622-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg622-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg622-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg622-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg622-server: oleg622-server.virtnet: executing set_default_debug -1 all pdsh@oleg622-client: oleg622-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg622-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg622-server: oleg622-server.virtnet: executing set_default_debug -1 all pdsh@oleg622-client: oleg622-server: ssh exited with exit code 1 Started lustre-OST0000 oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg622-client.virtnet: -o user_xattr,flock 192.168.206.122@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 140 (time 1788471688.81 total 10.00 last 14.00) - open/close 319 (time 1788471699.27 total 20.46 last 17.11) - open/close 603 (time 1788471709.30 total 30.49 last 28.31) - open/close 835 (time 1788471719.32 total 40.51 last 23.16) - open/close 1071 (time 1788471729.32 total 50.51 last 23.60) - open/close 1304 (time 1788471739.33 total 60.52 last 23.29) - open/close 1545 (time 1788471749.35 total 70.54 last 24.06) - open/close 1830 (time 1788471760.08 total 81.26 last 26.57) - open/close 2037 (time 1788471770.25 total 91.44 last 20.34) - open/close 2272 (time 1788471780.49 total 101.68 last 22.96) - open/close 2560 (time 1788471790.49 total 111.68 last 28.80) - open/close 2798 (time 1788471800.65 total 121.84 last 23.43) - open/close 3022 (time 1788471812.87 total 134.06 last 18.33) - open/close 3270 (time 1788471823.48 total 144.67 last 23.37) - open/close 3515 (time 1788471833.48 total 154.67 last 24.49) - open/close 3756 (time 1788471843.68 total 164.86 last 23.64) - open/close 3950 (time 1788471854.35 total 175.54 last 18.18) - open/close 4166 (time 1788471864.36 total 185.55 last 21.57) - open/close 4384 (time 1788471874.37 total 195.56 last 21.78) - open/close 4663 (time 1788471885.39 total 206.58 last 25.31) - open/close 4824 (time 1788471895.42 total 216.60 last 16.07) - open/close 5050 (time 1788471906.00 total 227.18 last 21.36) - open/close 5317 (time 1788471916.79 total 237.98 last 24.74) - open/close 5508 (time 1788471927.00 total 248.19 last 18.70) - open/close 5751 (time 1788471937.01 total 258.20 last 24.28) - open/close 5980 (time 1788471947.40 total 268.59 last 22.05) - open/close 6239 (time 1788471958.43 total 279.62 last 23.48) - open/close 6472 (time 1788471969.15 total 290.34 last 21.72) - open/close 6615 (time 1788471979.55 total 300.74 last 13.76) - open/close 6770 (time 1788471990.47 total 311.66 last 14.19) - open/close 6943 (time 1788472001.37 total 322.56 last 15.87) - open/close 7180 (time 1788472012.38 total 333.56 last 21.54) - open/close 7501 (time 1788472022.40 total 343.59 last 32.03) - open/close 7718 (time 1788472032.79 total 353.98 last 20.87) - open/close 7831 (time 1788472042.95 total 364.14 last 11.13) - open/close 8106 (time 1788472053.65 total 374.84 last 25.69) - open/close 8323 (time 1788472063.67 total 384.86 last 21.67) - open/close 8453 (time 1788472073.67 total 394.86 last 12.99) - open/close 8591 (time 1788472084.76 total 405.95 last 12.45) - open/close 8707 (time 1788472095.12 total 416.30 last 11.20) - open/close 8813 (time 1788472105.30 total 426.49 last 10.41) - open/close 8928 (time 1788472115.41 total 436.59 last 11.38) - open/close 9133 (time 1788472125.71 total 446.90 last 19.89) - open/close 9386 (time 1788472138.77 total 459.96 last 19.37) - open/close 9560 (time 1788472149.77 total 470.96 last 15.81) - open/close 9646 (time 1788472160.27 total 481.46 last 8.20) - open/close 9851 (time 1788472171.35 total 492.53 last 18.50) - open/close 10000 (time 1788472177.05 total 498.23 last 26.14) - open/close 10252 (time 1788472187.06 total 508.24 last 25.18) - open/close 10401 (time 1788472197.06 total 518.25 last 14.90) - open/close 10617 (time 1788472209.61 total 530.80 last 17.21) - open/close 10751 (time 1788472219.72 total 540.91 last 13.25) - open/close 11040 (time 1788472230.43 total 551.61 last 27.01) - open/close 11066 (time 1788472240.44 total 561.62 last 2.60) - open/close 11250 (time 1788472250.44 total 571.63 last 18.39) - open/close 11445 (time 1788472260.50 total 581.69 last 19.38) - open/close 11619 (time 1788472270.54 total 591.73 last 17.34) - open/close 11765 (time 1788472280.92 total 602.11 last 14.07) - open/close 11831 (time 1788472293.15 total 614.33 last 5.40) - open/close 11981 (time 1788472304.91 total 626.10 last 12.75) - open/close 12168 (time 1788472314.93 total 636.12 last 18.67) - open/close 12334 (time 1788472327.36 total 648.55 last 13.35) - open/close 12558 (time 1788472339.28 total 660.47 last 18.79) - open/close 12832 (time 1788472349.30 total 670.48 last 27.36) - open/close 13068 (time 1788472360.35 total 681.54 last 21.35) - open/close 13240 (time 1788472370.90 total 692.08 last 16.31) - open/close 13506 (time 1788472380.90 total 702.08 last 26.60) - open/close 13706 (time 1788472390.90 total 712.09 last 19.98) - open/close 13936 (time 1788472401.19 total 722.38 last 22.35) - open/close 14238 (time 1788472412.12 total 733.31 last 27.63) - open/close 14476 (time 1788472422.56 total 743.74 last 22.81) - open/close 14550 (time 1788472432.57 total 753.76 last 7.39) - open/close 14719 (time 1788472443.34 total 764.53 last 15.70) - open/close 14848 (time 1788472453.41 total 774.60 last 12.81) - open/close 15108 (time 1788472463.41 total 784.60 last 25.99) - open/close 15309 (time 1788472473.64 total 794.83 last 19.65) - open/close 15512 (time 1788472483.92 total 805.11 last 19.75) - open/close 15771 (time 1788472494.60 total 815.79 last 24.26) - open/close 16013 (time 1788472505.36 total 826.55 last 22.49) - open/close 16270 (time 1788472515.36 total 836.55 last 25.69) - open/close 16433 (time 1788472525.78 total 846.96 last 15.66) - open/close 16696 (time 1788472535.88 total 857.07 last 26.03) - open/close 16964 (time 1788472546.90 total 868.09 last 24.31) - open/close 17172 (time 1788472556.94 total 878.13 last 20.73) - open/close 17425 (time 1788472566.96 total 888.15 last 25.24) - open/close 17685 (time 1788472577.87 total 899.06 last 23.83) - open/close 17907 (time 1788472590.29 total 911.48 last 17.88) - open/close 18060 (time 1788472601.32 total 922.51 last 13.87) - open/close 18245 (time 1788472612.06 total 933.25 last 17.23) - open/close 18436 (time 1788472622.87 total 944.06 last 17.67) - open/close 18721 (time 1788472632.88 total 954.07 last 28.48) - open/close 18906 (time 1788472643.81 total 965.00 last 16.92) - open/close 19488 (time 1788472653.82 total 975.01 last 58.19) - open/close 20000 (time 1788472661.88 total 983.07 last 63.52) - open/close 20730 (time 1788472671.88 total 993.07 last 72.97) - open/close 21348 (time 1788472681.88 total 1003.07 last 61.79) - open/close 22018 (time 1788472691.90 total 1013.09 last 66.89) - open/close 22678 (time 1788472701.91 total 1023.10 last 65.91) - open/close 23326 (time 1788472711.93 total 1033.12 last 64.70) - open/close 24080 (time 1788472721.94 total 1043.13 last 75.32) - open/close 24718 (time 1788472731.95 total 1053.13 last 63.75) - open/close 25413 (time 1788472741.96 total 1063.14 last 69.43) - open/close 26034 (time 1788472751.96 total 1073.15 last 62.08) - open/close 26577 (time 1788472761.97 total 1083.16 last 54.23) - open/close 27146 (time 1788472771.98 total 1093.16 last 56.87) - open/close 27772 (time 1788472781.99 total 1103.18 last 62.49) - open/close 28429 (time 1788472792.01 total 1113.20 last 65.61) - open/close 29145 (time 1788472802.02 total 1123.21 last 71.54) - open/close 29858 (time 1788472812.03 total 1133.22 last 71.22) - open/close 30000 (time 1788472814.25 total 1135.44 last 63.96) - open/close 30617 (time 1788472824.25 total 1145.44 last 61.68) - open/close 31309 (time 1788472834.25 total 1155.44 last 69.18) - open/close 31979 (time 1788472844.26 total 1165.45 last 66.97) - open/close 32713 (time 1788472854.26 total 1175.45 last 73.37) - open/close 33431 (time 1788472864.27 total 1185.46 last 71.77) - open/close 34129 (time 1788472874.30 total 1195.49 last 69.57) - open/close 34810 (time 1788472884.31 total 1205.50 last 68.06) - open/close 35521 (time 1788472894.31 total 1215.50 last 71.07) - open/close 36228 (time 1788472904.31 total 1225.50 last 70.70) - open/close 36853 (time 1788472914.31 total 1235.50 last 62.48) - open/close 37479 (time 1788472924.32 total 1245.50 last 62.59) - open/close 38147 (time 1788472934.32 total 1255.51 last 66.75) - open/close 38845 (time 1788472944.34 total 1265.53 last 69.69) - open/close 39501 (time 1788472954.35 total 1275.54 last 65.52) - open/close 40000 (time 1788472962.13 total 1283.32 last 64.17) - open/close 40648 (time 1788472972.14 total 1293.33 last 64.71) - open/close 41337 (time 1788472982.14 total 1303.33 last 68.89) - open/close 42029 (time 1788472992.15 total 1313.34 last 69.16) - open/close 42744 (time 1788473002.20 total 1323.39 last 71.11) - open/close 43425 (time 1788473012.21 total 1333.39 last 68.08) - open/close 44082 (time 1788473022.22 total 1343.41 last 65.62) - open/close 44830 (time 1788473032.23 total 1353.42 last 74.71) - open/close 45553 (time 1788473042.24 total 1363.42 last 72.25) - open/close 46282 (time 1788473052.24 total 1373.43 last 72.85) - open/close 46991 (time 1788473062.25 total 1383.44 last 70.83) - open/close 47714 (time 1788473072.27 total 1393.46 last 72.18) - open/close 48450 (time 1788473082.28 total 1403.47 last 73.50) - open/close 49202 (time 1788473092.29 total 1413.48 last 75.14) - open/close 49939 (time 1788473102.30 total 1423.48 last 73.65) total: 50000 open/close in 1424.34 seconds: 35.10 ops/second - unlinked 0 (time 1788473107 ; total 1 ; last 1) - unlinked 10000 (time 1788473261 ; total 155 ; last 154) - unlinked 20000 (time 1788473426 ; total 320 ; last 165) - unlinked 30000 (time 1788473587 ; total 481 ; last 161) - unlinked 40000 (time 1788473745 ; total 639 ; last 158) total: 50000 unlinks in 805 seconds: 62.111801 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg622-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg622-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg622-server stop mds service on oleg622-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg622-server unloading modules via unload_modules_local on: 'oleg622-server' oleg622-server: oleg622-server.virtnet: executing unload_modules_local oleg622-server: kmemleak disabled modules unloaded. pdsh@oleg622-client: oleg622-client: ssh exited with exit code 2 pdsh@oleg622-client: oleg622-server: ssh exited with exit code 2 pdsh@oleg622-client: oleg622-client: ssh exited with exit code 2 pdsh@oleg622-client: oleg622-server: ssh exited with exit code 2