== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 15:33:50 (1789068830) start mds service on oleg445-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1931560 Sep 10 03:22 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: kmemleak disabled oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions oleg445-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg445-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg445-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg445-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg445-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-OST0000 oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 186 (time 1789068932.35 total 10.11 last 18.40) - open/close 256 (time 1789068942.56 total 20.32 last 6.86) - open/close 367 (time 1789068953.31 total 31.07 last 10.32) - open/close 447 (time 1789068963.71 total 41.47 last 7.70) - open/close 597 (time 1789068974.34 total 52.10 last 14.10) - open/close 688 (time 1789068984.71 total 62.47 last 8.78) - open/close 856 (time 1789068995.60 total 73.35 last 15.43) - open/close 1013 (time 1789069006.42 total 84.18 last 14.50) - open/close 1188 (time 1789069017.14 total 94.90 last 16.33) - open/close 1282 (time 1789069027.84 total 105.60 last 8.78) - open/close 1516 (time 1789069037.85 total 115.61 last 23.38) - open/close 1667 (time 1789069049.35 total 127.11 last 13.14) - open/close 1812 (time 1789069059.52 total 137.28 last 14.25) - open/close 2008 (time 1789069069.95 total 147.70 last 18.81) - open/close 2145 (time 1789069080.40 total 158.16 last 13.11) - open/close 2259 (time 1789069090.40 total 168.16 last 11.39) - open/close 2388 (time 1789069100.79 total 178.55 last 12.42) - open/close 2602 (time 1789069110.80 total 188.56 last 21.38) - open/close 2742 (time 1789069120.92 total 198.68 last 13.83) - open/close 2866 (time 1789069130.98 total 208.73 last 12.33) - open/close 2978 (time 1789069141.90 total 219.66 last 10.25) - open/close 3222 (time 1789069151.92 total 229.68 last 24.36) - open/close 3442 (time 1789069161.93 total 239.68 last 21.98) - open/close 3665 (time 1789069172.38 total 250.14 last 21.33) - open/close 3887 (time 1789069182.38 total 260.14 last 22.19) - open/close 4003 (time 1789069197.09 total 274.85 last 7.89) - open/close 4181 (time 1789069207.48 total 285.24 last 17.14) - open/close 4338 (time 1789069217.57 total 295.33 last 15.56) - open/close 4400 (time 1789069228.62 total 306.37 last 5.61) - open/close 4573 (time 1789069238.62 total 316.38 last 17.29) - open/close 4731 (time 1789069249.21 total 326.97 last 14.92) - open/close 4803 (time 1789069259.24 total 337.00 last 7.18) - open/close 4912 (time 1789069270.03 total 347.79 last 10.11) - open/close 5039 (time 1789069281.09 total 358.84 last 11.49) - open/close 5276 (time 1789069291.13 total 368.89 last 23.60) - open/close 5482 (time 1789069304.13 total 381.89 last 15.85) - open/close 5571 (time 1789069314.16 total 391.92 last 8.87) - open/close 5639 (time 1789069324.22 total 401.97 last 6.76) - open/close 5811 (time 1789069336.96 total 414.71 last 13.50) - open/close 5937 (time 1789069346.98 total 424.74 last 12.57) - open/close 6050 (time 1789069360.00 total 437.76 last 8.68) - open/close 6136 (time 1789069370.47 total 448.23 last 8.22) - open/close 6218 (time 1789069381.52 total 459.28 last 7.42) - open/close 6319 (time 1789069391.90 total 469.66 last 9.73) - open/close 6497 (time 1789069401.92 total 479.68 last 17.76) - open/close 6641 (time 1789069414.89 total 492.65 last 11.10) - open/close 6769 (time 1789069425.12 total 502.88 last 12.51) - open/close 6909 (time 1789069435.12 total 512.88 last 13.99) - open/close 7001 (time 1789069445.14 total 522.90 last 9.19) - open/close 7140 (time 1789069455.44 total 533.20 last 13.49) - open/close 7257 (time 1789069466.37 total 544.13 last 10.71) - open/close 7423 (time 1789069478.29 total 556.05 last 13.93) - open/close 7499 (time 1789069489.29 total 567.05 last 6.91) - open/close 7609 (time 1789069499.30 total 577.05 last 10.99) - open/close 7734 (time 1789069509.31 total 587.07 last 12.48) - open/close 7869 (time 1789069519.82 total 597.58 last 12.84) - open/close 7957 (time 1789069530.24 total 608.00 last 8.45) - open/close 8046 (time 1789069540.81 total 618.57 last 8.42) - open/close 8182 (time 1789069551.30 total 629.06 last 12.97) - open/close 8432 (time 1789069562.05 total 639.81 last 23.24) - open/close 8592 (time 1789069572.06 total 649.82 last 15.99) - open/close 8778 (time 1789069582.11 total 659.87 last 18.50) - open/close 8908 (time 1789069592.19 total 669.95 last 12.90) - open/close 9085 (time 1789069602.21 total 679.97 last 17.67) - open/close 9247 (time 1789069612.24 total 690.00 last 16.15) - open/close 9390 (time 1789069623.31 total 701.07 last 12.92) - open/close 9558 (time 1789069634.41 total 712.17 last 15.14) - open/close 9623 (time 1789069645.07 total 722.83 last 6.10) - open/close 9730 (time 1789069655.09 total 732.85 last 10.68) - open/close 9886 (time 1789069665.80 total 743.56 last 14.56) - open/close 10000 (time 1789069672.32 total 750.08 last 17.48) - open/close 10181 (time 1789069682.37 total 760.12 last 18.02) - open/close 10305 (time 1789069692.37 total 770.13 last 12.39) - open/close 10532 (time 1789069703.03 total 780.79 last 21.29) - open/close 10616 (time 1789069715.52 total 793.28 last 6.73) - open/close 10796 (time 1789069725.63 total 803.39 last 17.81) - open/close 10974 (time 1789069737.68 total 815.44 last 14.77) - open/close 11170 (time 1789069747.70 total 825.46 last 19.56) - open/close 11295 (time 1789069758.02 total 835.78 last 12.12) - open/close 11408 (time 1789069768.44 total 846.20 last 10.84) - open/close 11523 (time 1789069778.45 total 856.21 last 11.48) - open/close 11628 (time 1789069788.48 total 866.23 last 10.48) - open/close 11727 (time 1789069799.17 total 876.93 last 9.26) - open/close 11928 (time 1789069809.87 total 887.62 last 18.80) - open/close 12041 (time 1789069819.97 total 897.73 last 11.18) - open/close 12118 (time 1789069829.99 total 907.75 last 7.68) - open/close 12277 (time 1789069840.04 total 917.79 last 15.84) - open/close 12431 (time 1789069852.86 total 930.62 last 12.01) - open/close 12600 (time 1789069863.62 total 941.38 last 15.71) - open/close 12763 (time 1789069873.63 total 951.39 last 16.27) - open/close 12967 (time 1789069886.10 total 963.85 last 16.37) - open/close 13211 (time 1789069896.10 total 973.86 last 24.39) - open/close 13292 (time 1789069907.52 total 985.28 last 7.09) - open/close 13385 (time 1789069919.05 total 996.81 last 8.07) - open/close 13445 (time 1789069929.08 total 1006.83 last 5.98) - open/close 13596 (time 1789069939.08 total 1016.84 last 15.09) - open/close 13704 (time 1789069949.99 total 1027.75 last 9.90) - open/close 13941 (time 1789069960.02 total 1037.78 last 23.63) - open/close 14085 (time 1789069970.03 total 1047.79 last 14.39) - open/close 14222 (time 1789069980.70 total 1058.46 last 12.84) - open/close 14317 (time 1789069992.65 total 1070.41 last 7.95) - open/close 14419 (time 1789070002.67 total 1080.43 last 10.18) - open/close 14485 (time 1789070012.69 total 1090.45 last 6.59) - open/close 14540 (time 1789070022.72 total 1100.47 last 5.49) - open/close 14787 (time 1789070032.73 total 1110.49 last 24.67) - open/close 14895 (time 1789070042.74 total 1120.50 last 10.79) - open/close 15212 (time 1789070052.75 total 1130.50 last 31.68) - open/close 15766 (time 1789070062.75 total 1140.51 last 55.37) - open/close 16346 (time 1789070072.76 total 1150.52 last 57.94) - open/close 16908 (time 1789070082.77 total 1160.53 last 56.13) - open/close 17417 (time 1789070092.77 total 1170.53 last 50.90) - open/close 17985 (time 1789070102.77 total 1180.53 last 56.80) - open/close 18554 (time 1789070112.78 total 1190.54 last 56.85) - open/close 19210 (time 1789070122.79 total 1200.55 last 65.57) - open/close 19871 (time 1789070132.81 total 1210.56 last 65.98) - open/close 20000 (time 1789070135.18 total 1212.94 last 54.27) - open/close 20608 (time 1789070145.19 total 1222.95 last 60.74) - open/close 21265 (time 1789070155.20 total 1232.95 last 65.67) - open/close 21778 (time 1789070165.20 total 1242.96 last 51.26) - open/close 22415 (time 1789070175.22 total 1252.97 last 63.61) - open/close 23030 (time 1789070185.25 total 1263.00 last 61.32) - open/close 23652 (time 1789070195.25 total 1273.01 last 62.16) - open/close 24264 (time 1789070205.26 total 1283.02 last 61.13) - open/close 24944 (time 1789070215.29 total 1293.05 last 67.80) - open/close 25600 (time 1789070225.31 total 1303.07 last 65.50) - open/close 26237 (time 1789070235.32 total 1313.08 last 63.64) - open/close 26835 (time 1789070245.33 total 1323.09 last 59.72) - open/close 27409 (time 1789070255.35 total 1333.11 last 57.31) - open/close 28003 (time 1789070265.35 total 1343.11 last 59.39) - open/close 28557 (time 1789070275.36 total 1353.11 last 55.36) - open/close 29122 (time 1789070285.37 total 1363.12 last 56.44) - open/close 29677 (time 1789070295.38 total 1373.13 last 55.44) - open/close 30000 (time 1789070300.38 total 1378.14 last 64.50) - open/close 30531 (time 1789070310.39 total 1388.15 last 53.05) - open/close 31107 (time 1789070320.40 total 1398.16 last 57.58) - open/close 31699 (time 1789070330.40 total 1408.16 last 59.16) - open/close 32221 (time 1789070340.41 total 1418.17 last 52.16) - open/close 32791 (time 1789070350.42 total 1428.18 last 56.94) - open/close 33329 (time 1789070360.43 total 1438.19 last 53.73) - open/close 33849 (time 1789070370.44 total 1448.20 last 51.99) - open/close 34464 (time 1789070380.44 total 1458.20 last 61.48) - open/close 35152 (time 1789070390.45 total 1468.21 last 68.72) - open/close 35837 (time 1789070400.46 total 1478.21 last 68.48) - open/close 36470 (time 1789070410.47 total 1488.23 last 63.21) - open/close 37124 (time 1789070420.48 total 1498.24 last 65.32) - open/close 37857 (time 1789070430.49 total 1508.25 last 73.22) - open/close 38517 (time 1789070440.50 total 1518.26 last 65.95) - open/close 39145 (time 1789070450.50 total 1528.26 last 62.79) - open/close 39767 (time 1789070460.51 total 1538.27 last 62.16) - open/close 40000 (time 1789070465.40 total 1543.15 last 47.68) - open/close 40691 (time 1789070475.41 total 1553.17 last 68.97) - open/close 41347 (time 1789070485.41 total 1563.17 last 65.60) - open/close 42013 (time 1789070495.42 total 1573.18 last 66.56) - open/close 42556 (time 1789070505.43 total 1583.18 last 54.27) - open/close 43145 (time 1789070515.43 total 1593.19 last 58.85) - open/close 43662 (time 1789070525.45 total 1603.21 last 51.63) - open/close 44113 (time 1789070535.47 total 1613.22 last 45.02) - open/close 44746 (time 1789070545.47 total 1623.22 last 63.29) - open/close 45348 (time 1789070555.48 total 1633.24 last 60.12) - open/close 45935 (time 1789070565.49 total 1643.24 last 58.67) - open/close 46548 (time 1789070575.50 total 1653.26 last 61.20) - open/close 47036 (time 1789070585.50 total 1663.26 last 48.79) - open/close 47488 (time 1789070595.53 total 1673.29 last 45.09) - open/close 48087 (time 1789070605.54 total 1683.30 last 59.82) - open/close 48652 (time 1789070615.54 total 1693.30 last 56.48) - open/close 49240 (time 1789070625.55 total 1703.31 last 58.76) - open/close 49711 (time 1789070635.56 total 1713.32 last 47.06) total: 50000 open/close in 1719.13 seconds: 29.08 ops/second - unlinked 0 (time 1789070648 ; total 0 ; last 0) - unlinked 10000 (time 1789070824 ; total 176 ; last 176) - unlinked 20000 (time 1789071007 ; total 359 ; last 183) - unlinked 30000 (time 1789071192 ; total 544 ; last 185) - unlinked 40000 (time 1789071371 ; total 723 ; last 179) total: 50000 unlinks in 893 seconds: 55.991043 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg445-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg445-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg445-server stop mds service on oleg445-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg445-server unloading modules via unload_modules_local on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing unload_modules_local oleg445-server: kmemleak disabled modules unloaded. pdsh@oleg445-client: oleg445-client: ssh exited with exit code 2 pdsh@oleg445-client: oleg445-server: ssh exited with exit code 2 pdsh@oleg445-client: oleg445-client: ssh exited with exit code 2 pdsh@oleg445-client: oleg445-server: ssh exited with exit code 2