== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:47:38 (1788216458) start mds service on oleg140-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1755536 Aug 31 10:08 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg140-server' oleg140-server: oleg140-server.virtnet: executing load_modules_local oleg140-server: kmemleak disabled oleg140-server: Loading modules from /home/green/git/lustre-release/lustre oleg140-server: detected 4 online CPUs by sysfs oleg140-server: MODOPTS_LIBCFS= oleg140-server: Force libcfs to create 2 CPU partitions oleg140-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg140-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg140-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg140-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg140-server Start mds2: mount -t lustre -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg140-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-OST0000 oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg140-client.virtnet: -o user_xattr,flock 192.168.201.140@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 224 (time 1788216575.42 total 10.41 last 21.51) - open/close 453 (time 1788216588.13 total 23.12 last 18.02) - open/close 667 (time 1788216599.09 total 34.08 last 19.53) - open/close 898 (time 1788216609.45 total 44.44 last 22.29) - open/close 1022 (time 1788216619.47 total 54.46 last 12.37) - open/close 1152 (time 1788216629.61 total 64.60 last 12.83) - open/close 1376 (time 1788216644.35 total 79.34 last 15.19) - open/close 1635 (time 1788216654.36 total 89.36 last 25.87) - open/close 1790 (time 1788216664.68 total 99.67 last 15.02) - open/close 2031 (time 1788216674.70 total 109.69 last 24.06) - open/close 2202 (time 1788216684.70 total 119.70 last 17.09) - open/close 2410 (time 1788216695.15 total 130.15 last 19.90) - open/close 2605 (time 1788216705.58 total 140.57 last 18.71) - open/close 2887 (time 1788216715.59 total 150.58 last 28.18) - open/close 2988 (time 1788216725.80 total 160.79 last 9.89) - open/close 3248 (time 1788216735.81 total 170.81 last 25.97) - open/close 3423 (time 1788216746.80 total 181.79 last 15.94) - open/close 3656 (time 1788216756.80 total 191.80 last 23.28) - open/close 3779 (time 1788216766.94 total 201.93 last 12.13) - open/close 4018 (time 1788216776.96 total 211.95 last 23.86) - open/close 4238 (time 1788216786.96 total 221.95 last 21.99) - open/close 4444 (time 1788216796.97 total 231.96 last 20.58) - open/close 4534 (time 1788216807.61 total 242.61 last 8.45) - open/close 4660 (time 1788216817.91 total 252.90 last 12.24) - open/close 4855 (time 1788216828.01 total 263.00 last 19.31) - open/close 5059 (time 1788216838.02 total 273.02 last 20.36) - open/close 5145 (time 1788216848.23 total 283.22 last 8.42) - open/close 5228 (time 1788216859.10 total 294.10 last 7.64) - open/close 5374 (time 1788216869.10 total 304.10 last 14.60) - open/close 5501 (time 1788216879.59 total 314.58 last 12.11) - open/close 5724 (time 1788216889.61 total 324.60 last 22.25) - open/close 5986 (time 1788216900.08 total 335.07 last 25.03) - open/close 6204 (time 1788216910.70 total 345.69 last 20.53) - open/close 6464 (time 1788216920.70 total 355.69 last 26.00) - open/close 6704 (time 1788216931.38 total 366.37 last 22.46) - open/close 6935 (time 1788216941.79 total 376.78 last 22.19) - open/close 7151 (time 1788216951.80 total 386.79 last 21.58) - open/close 7271 (time 1788216962.53 total 397.52 last 11.19) - open/close 7405 (time 1788216972.79 total 407.78 last 13.06) - open/close 7671 (time 1788216983.40 total 418.39 last 25.07) - open/close 7926 (time 1788216994.03 total 429.02 last 23.99) - open/close 8148 (time 1788217004.67 total 439.66 last 20.87) - open/close 8363 (time 1788217014.83 total 449.83 last 21.15) - open/close 8559 (time 1788217024.84 total 459.83 last 19.59) - open/close 8785 (time 1788217034.85 total 469.84 last 22.57) - open/close 8889 (time 1788217044.85 total 479.84 last 10.40) - open/close 9073 (time 1788217055.61 total 490.60 last 17.10) - open/close 9324 (time 1788217065.63 total 500.62 last 25.07) - open/close 9567 (time 1788217075.63 total 510.62 last 24.30) - open/close 9761 (time 1788217086.17 total 521.16 last 18.40) - open/close 9996 (time 1788217096.19 total 531.18 last 23.47) - open/close 10164 (time 1788217106.20 total 541.19 last 16.78) - open/close 10344 (time 1788217117.09 total 552.09 last 16.52) - open/close 10566 (time 1788217127.10 total 562.09 last 22.19) - open/close 10781 (time 1788217137.70 total 572.70 last 20.27) - open/close 10963 (time 1788217149.23 total 584.23 last 15.78) - open/close 11163 (time 1788217159.24 total 594.23 last 19.99) - open/close 11445 (time 1788217169.84 total 604.83 last 26.60) - open/close 11707 (time 1788217179.84 total 614.84 last 26.19) - open/close 11870 (time 1788217191.40 total 626.39 last 14.11) - open/close 12003 (time 1788217201.41 total 636.41 last 13.28) - open/close 12081 (time 1788217211.56 total 646.55 last 7.69) - open/close 12297 (time 1788217222.57 total 657.56 last 19.62) - open/close 12423 (time 1788217232.57 total 667.57 last 12.59) - open/close 12652 (time 1788217243.57 total 678.56 last 20.83) - open/close 12869 (time 1788217254.19 total 689.18 last 20.43) - open/close 13038 (time 1788217264.19 total 699.18 last 16.90) - open/close 13191 (time 1788217274.42 total 709.41 last 14.95) - open/close 13386 (time 1788217284.61 total 719.60 last 19.13) - open/close 13618 (time 1788217295.21 total 730.20 last 21.89) - open/close 13876 (time 1788217305.27 total 740.26 last 25.65) - open/close 14088 (time 1788217315.62 total 750.61 last 20.47) - open/close 14334 (time 1788217326.30 total 761.29 last 23.03) - open/close 14564 (time 1788217337.29 total 772.28 last 20.93) - open/close 14766 (time 1788217347.31 total 782.30 last 20.16) - open/close 15010 (time 1788217358.06 total 793.06 last 22.69) - open/close 15220 (time 1788217368.12 total 803.11 last 20.89) - open/close 15386 (time 1788217378.13 total 813.12 last 16.57) - open/close 15627 (time 1788217390.95 total 825.94 last 18.81) - open/close 15825 (time 1788217400.97 total 835.96 last 19.75) - open/close 15990 (time 1788217411.69 total 846.68 last 15.40) - open/close 16241 (time 1788217421.70 total 856.69 last 25.07) - open/close 16480 (time 1788217432.33 total 867.32 last 22.48) - open/close 16689 (time 1788217442.38 total 877.37 last 20.81) - open/close 16879 (time 1788217452.40 total 887.40 last 18.94) - open/close 17147 (time 1788217462.41 total 897.40 last 26.80) - open/close 17348 (time 1788217472.42 total 907.41 last 20.07) - open/close 17494 (time 1788217482.98 total 917.97 last 13.83) - open/close 17705 (time 1788217493.81 total 928.81 last 19.48) - open/close 17851 (time 1788217503.92 total 938.92 last 14.44) - open/close 18036 (time 1788217514.65 total 949.64 last 17.24) - open/close 18261 (time 1788217524.67 total 959.67 last 22.45) - open/close 18499 (time 1788217535.27 total 970.26 last 22.46) - open/close 18849 (time 1788217545.27 total 980.26 last 35.00) - open/close 19696 (time 1788217555.27 total 990.26 last 84.68) - open/close 20000 (time 1788217559.11 total 994.11 last 79.10) - open/close 20817 (time 1788217569.12 total 1004.12 last 81.61) - open/close 21507 (time 1788217579.13 total 1014.13 last 68.94) - open/close 22280 (time 1788217589.14 total 1024.14 last 77.22) - open/close 22993 (time 1788217599.15 total 1034.14 last 71.26) - open/close 23782 (time 1788217609.16 total 1044.15 last 78.83) - open/close 24569 (time 1788217619.16 total 1054.15 last 78.66) - open/close 25300 (time 1788217629.16 total 1064.16 last 73.09) - open/close 26092 (time 1788217639.18 total 1074.17 last 79.11) - open/close 26909 (time 1788217649.20 total 1084.19 last 81.49) - open/close 27622 (time 1788217659.21 total 1094.20 last 71.24) - open/close 28430 (time 1788217669.21 total 1104.21 last 80.76) - open/close 29167 (time 1788217679.22 total 1114.22 last 73.64) - open/close 29837 (time 1788217689.23 total 1124.22 last 66.98) - open/close 30000 (time 1788217691.37 total 1126.36 last 76.09) - open/close 30806 (time 1788217701.39 total 1136.38 last 80.46) - open/close 31492 (time 1788217711.39 total 1146.38 last 68.56) - open/close 32084 (time 1788217721.40 total 1156.39 last 59.14) - open/close 32665 (time 1788217731.43 total 1166.43 last 57.91) - open/close 33293 (time 1788217741.45 total 1176.44 last 62.69) - open/close 33947 (time 1788217751.46 total 1186.45 last 65.35) - open/close 34676 (time 1788217761.47 total 1196.46 last 72.84) - open/close 35436 (time 1788217771.47 total 1206.47 last 75.95) - open/close 36237 (time 1788217781.48 total 1216.47 last 80.06) - open/close 36943 (time 1788217791.48 total 1226.48 last 70.57) - open/close 37585 (time 1788217801.49 total 1236.48 last 64.19) - open/close 38276 (time 1788217811.49 total 1246.48 last 69.05) - open/close 38982 (time 1788217821.50 total 1256.49 last 70.55) - open/close 39622 (time 1788217831.52 total 1266.51 last 63.90) - open/close 40000 (time 1788217836.50 total 1271.49 last 75.88) - open/close 40807 (time 1788217846.50 total 1281.50 last 80.64) - open/close 41616 (time 1788217856.52 total 1291.52 last 80.73) - open/close 42427 (time 1788217866.53 total 1301.52 last 81.06) - open/close 43325 (time 1788217876.53 total 1311.53 last 89.77) - open/close 44078 (time 1788217886.54 total 1321.53 last 75.26) - open/close 44743 (time 1788217896.55 total 1331.55 last 66.40) - open/close 45455 (time 1788217906.58 total 1341.57 last 71.03) - open/close 46100 (time 1788217916.58 total 1351.58 last 64.46) - open/close 46863 (time 1788217926.59 total 1361.58 last 76.28) - open/close 47721 (time 1788217936.59 total 1371.58 last 85.78) - open/close 48405 (time 1788217946.59 total 1381.59 last 68.37) - open/close 49129 (time 1788217956.60 total 1391.59 last 72.35) - open/close 49906 (time 1788217966.60 total 1401.60 last 77.67) total: 50000 open/close in 1402.71 seconds: 35.65 ops/second - unlinked 0 (time 1788217971 ; total 0 ; last 0) - unlinked 10000 (time 1788218111 ; total 140 ; last 140) - unlinked 20000 (time 1788218255 ; total 284 ; last 144) - unlinked 30000 (time 1788218390 ; total 419 ; last 135) - unlinked 40000 (time 1788218523 ; total 552 ; last 133) total: 50000 unlinks in 693 seconds: 72.150070 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg140-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg140-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg140-server stop mds service on oleg140-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg140-server stop mds service on oleg140-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg140-server unloading modules via unload_modules_local on: 'oleg140-server' oleg140-server: oleg140-server.virtnet: executing unload_modules_local oleg140-server: kmemleak disabled modules unloaded. pdsh@oleg140-client: oleg140-client: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-server: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-client: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-server: ssh exited with exit code 2