== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:12:10 (1781237530) start mds service on oleg328-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg328-server: detected 4 online CPUs by sysfs oleg328-server: MODOPTS_LIBCFS= oleg328-server: Force libcfs to create 2 CPU partitions oleg328-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg328-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg328-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg328-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Started lustre-OST0000 oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg328-client.virtnet: -o user_xattr,flock 192.168.203.128@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 189 (time 1781237613.09 total 10.99 last 17.20) - open/close 426 (time 1781237623.10 total 20.99 last 23.69) - open/close 622 (time 1781237633.44 total 31.34 last 18.94) - open/close 864 (time 1781237644.32 total 42.22 last 22.25) - open/close 1114 (time 1781237654.33 total 52.22 last 24.98) - open/close 1355 (time 1781237664.36 total 62.25 last 24.02) - open/close 1537 (time 1781237675.37 total 73.27 last 16.53) - open/close 1753 (time 1781237686.37 total 84.26 last 19.64) - open/close 1934 (time 1781237696.38 total 94.27 last 18.08) - open/close 2129 (time 1781237707.19 total 105.08 last 18.04) - open/close 2365 (time 1781237718.18 total 116.08 last 21.47) - open/close 2603 (time 1781237728.68 total 126.58 last 22.66) - open/close 2847 (time 1781237739.12 total 137.01 last 23.39) - open/close 3012 (time 1781237749.92 total 147.82 last 15.27) - open/close 3102 (time 1781237759.97 total 157.86 last 8.96) - open/close 3314 (time 1781237770.41 total 168.31 last 20.29) - open/close 3557 (time 1781237780.42 total 178.32 last 24.28) - open/close 3815 (time 1781237793.11 total 191.01 last 20.33) - open/close 3984 (time 1781237804.06 total 201.95 last 15.44) - open/close 4157 (time 1781237814.24 total 212.14 last 16.99) - open/close 4358 (time 1781237824.63 total 222.53 last 19.35) - open/close 4563 (time 1781237835.26 total 233.16 last 19.28) - open/close 4777 (time 1781237845.29 total 243.19 last 21.34) - open/close 4948 (time 1781237855.32 total 253.22 last 17.05) - open/close 5158 (time 1781237865.33 total 263.23 last 20.97) - open/close 5355 (time 1781237876.40 total 274.30 last 17.80) - open/close 5616 (time 1781237887.05 total 284.95 last 24.49) - open/close 5865 (time 1781237897.06 total 294.96 last 24.88) - open/close 5949 (time 1781237907.07 total 304.97 last 8.39) - open/close 6189 (time 1781237917.61 total 315.51 last 22.77) - open/close 6312 (time 1781237927.98 total 325.87 last 11.87) - open/close 6440 (time 1781237938.33 total 336.23 last 12.36) - open/close 6635 (time 1781237949.15 total 347.04 last 18.03) - open/close 6867 (time 1781237959.71 total 357.61 last 21.97) - open/close 7031 (time 1781237970.13 total 368.02 last 15.74) - open/close 7257 (time 1781237980.77 total 378.67 last 21.24) - open/close 7516 (time 1781237990.77 total 388.67 last 25.90) - open/close 7725 (time 1781238000.77 total 398.67 last 20.90) - open/close 7944 (time 1781238010.92 total 408.81 last 21.59) - open/close 8197 (time 1781238021.85 total 419.75 last 23.14) - open/close 8343 (time 1781238032.17 total 430.07 last 14.14) - open/close 8572 (time 1781238042.85 total 440.74 last 21.46) - open/close 8642 (time 1781238052.85 total 450.74 last 7.00) - open/close 8769 (time 1781238062.91 total 460.80 last 12.62) - open/close 8894 (time 1781238073.06 total 470.96 last 12.31) - open/close 9080 (time 1781238083.30 total 481.20 last 18.17) - open/close 9300 (time 1781238093.31 total 491.21 last 21.97) - open/close 9535 (time 1781238112.54 total 510.44 last 12.22) - open/close 9712 (time 1781238123.11 total 521.01 last 16.75) - open/close 9940 (time 1781238135.79 total 533.68 last 17.98) - open/close 10000 (time 1781238138.84 total 536.74 last 19.63) - open/close 10205 (time 1781238149.30 total 547.20 last 19.61) - open/close 10468 (time 1781238161.06 total 558.96 last 22.36) - open/close 10687 (time 1781238171.58 total 569.48 last 20.81) - open/close 10891 (time 1781238181.65 total 579.55 last 20.26) - open/close 11145 (time 1781238192.66 total 590.56 last 23.08) - open/close 11390 (time 1781238202.68 total 600.58 last 24.45) - open/close 11589 (time 1781238212.70 total 610.60 last 19.85) - open/close 11813 (time 1781238223.00 total 620.90 last 21.75) - open/close 12062 (time 1781238233.01 total 630.91 last 24.88) - open/close 12195 (time 1781238243.03 total 640.93 last 13.27) - open/close 12426 (time 1781238253.60 total 651.50 last 21.86) - open/close 12607 (time 1781238263.60 total 661.50 last 18.10) - open/close 12788 (time 1781238273.61 total 671.51 last 18.08) - open/close 12943 (time 1781238283.64 total 681.53 last 15.47) - open/close 13068 (time 1781238293.85 total 691.75 last 12.23) - open/close 13264 (time 1781238303.87 total 701.77 last 19.57) - open/close 13388 (time 1781238314.03 total 711.93 last 12.21) - open/close 13588 (time 1781238324.84 total 722.74 last 18.50) - open/close 13772 (time 1781238334.84 total 732.74 last 18.40) - open/close 14003 (time 1781238345.73 total 743.63 last 21.20) - open/close 14238 (time 1781238356.64 total 754.54 last 21.54) - open/close 14487 (time 1781238366.91 total 764.81 last 24.25) - open/close 14610 (time 1781238376.92 total 774.82 last 12.29) - open/close 14774 (time 1781238387.12 total 785.02 last 16.08) - open/close 14985 (time 1781238397.59 total 795.49 last 20.14) - open/close 15229 (time 1781238408.17 total 806.07 last 23.07) - open/close 15455 (time 1781238418.85 total 816.74 last 21.17) - open/close 15660 (time 1781238429.01 total 826.90 last 20.18) - open/close 15929 (time 1781238439.72 total 837.61 last 25.11) - open/close 16163 (time 1781238449.72 total 847.62 last 23.39) - open/close 16399 (time 1781238459.74 total 857.64 last 23.55) - open/close 16611 (time 1781238469.75 total 867.65 last 21.18) - open/close 16766 (time 1781238479.78 total 877.68 last 15.45) - open/close 16856 (time 1781238495.05 total 892.95 last 5.89) - open/close 17075 (time 1781238505.07 total 902.97 last 21.86) - open/close 17183 (time 1781238515.87 total 913.77 last 10.00) - open/close 17365 (time 1781238525.90 total 923.79 last 18.16) - open/close 17613 (time 1781238536.35 total 934.25 last 23.72) - open/close 17794 (time 1781238546.35 total 944.25 last 18.09) - open/close 18030 (time 1781238557.22 total 955.12 last 21.72) - open/close 18424 (time 1781238567.27 total 965.16 last 39.22) - open/close 19025 (time 1781238577.27 total 975.17 last 60.08) - open/close 19593 (time 1781238587.27 total 985.17 last 56.77) - open/close 20000 (time 1781238593.67 total 991.57 last 63.62) - open/close 20677 (time 1781238603.67 total 1001.57 last 67.69) - open/close 21293 (time 1781238613.68 total 1011.58 last 61.56) - open/close 21967 (time 1781238623.69 total 1021.58 last 67.35) - open/close 22672 (time 1781238633.70 total 1031.59 last 70.42) - open/close 23351 (time 1781238643.71 total 1041.60 last 67.84) - open/close 24016 (time 1781238653.71 total 1051.61 last 66.45) - open/close 24650 (time 1781238663.74 total 1061.64 last 63.23) - open/close 25295 (time 1781238673.75 total 1071.65 last 64.44) - open/close 25867 (time 1781238683.77 total 1081.67 last 57.06) - open/close 26541 (time 1781238693.77 total 1091.67 last 67.39) - open/close 27208 (time 1781238703.79 total 1101.69 last 66.59) - open/close 27815 (time 1781238713.79 total 1111.69 last 60.67) - open/close 28443 (time 1781238723.80 total 1121.70 last 62.77) - open/close 29073 (time 1781238733.81 total 1131.70 last 62.95) - open/close 29692 (time 1781238743.81 total 1141.71 last 61.86) - open/close 30000 (time 1781238748.60 total 1146.50 last 64.30) - open/close 30617 (time 1781238758.61 total 1156.51 last 61.64) - open/close 31237 (time 1781238768.62 total 1166.52 last 61.94) - open/close 31843 (time 1781238778.64 total 1176.54 last 60.50) - open/close 32474 (time 1781238788.64 total 1186.54 last 63.07) - open/close 33122 (time 1781238798.65 total 1196.55 last 64.75) - open/close 33772 (time 1781238808.66 total 1206.55 last 64.97) - open/close 34418 (time 1781238818.66 total 1216.56 last 64.58) - open/close 35114 (time 1781238828.68 total 1226.58 last 69.47) - open/close 35768 (time 1781238838.68 total 1236.58 last 65.36) - open/close 36450 (time 1781238848.69 total 1246.59 last 68.17) - open/close 37138 (time 1781238858.69 total 1256.59 last 68.78) - open/close 37808 (time 1781238868.70 total 1266.60 last 66.96) - open/close 38419 (time 1781238878.70 total 1276.60 last 61.08) - open/close 39053 (time 1781238888.72 total 1286.61 last 63.30) - open/close 39701 (time 1781238898.72 total 1296.62 last 64.78) - open/close 40000 (time 1781238903.06 total 1300.96 last 68.86) - open/close 40654 (time 1781238913.07 total 1310.96 last 65.38) - open/close 41232 (time 1781238923.07 total 1320.97 last 57.77) - open/close 41852 (time 1781238933.09 total 1330.99 last 61.89) - open/close 42448 (time 1781238943.10 total 1341.00 last 59.54) - open/close 43046 (time 1781238953.10 total 1351.00 last 59.78) - open/close 43647 (time 1781238963.11 total 1361.01 last 60.03) - open/close 44260 (time 1781238973.11 total 1371.01 last 61.30) - open/close 44897 (time 1781238983.12 total 1381.01 last 63.68) - open/close 45588 (time 1781238993.12 total 1391.02 last 69.09) - open/close 46257 (time 1781239003.13 total 1401.03 last 66.81) - open/close 46908 (time 1781239013.15 total 1411.05 last 64.99) - open/close 47590 (time 1781239023.16 total 1421.06 last 68.12) - open/close 48238 (time 1781239033.20 total 1431.10 last 64.52) - open/close 48907 (time 1781239043.21 total 1441.10 last 66.88) - open/close 49496 (time 1781239053.21 total 1451.11 last 58.87) total: 50000 open/close in 1459.35 seconds: 34.26 ops/second - unlinked 0 (time 1781239065 ; total 0 ; last 0) - unlinked 10000 (time 1781239226 ; total 161 ; last 161) - unlinked 20000 (time 1781239392 ; total 327 ; last 166) - unlinked 30000 (time 1781239569 ; total 504 ; last 177) - unlinked 40000 (time 1781239738 ; total 673 ; last 169) total: 50000 unlinks in 850 seconds: 58.823528 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg328-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg328-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg328-server stop mds service on oleg328-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg328-server unloading modules via unload_modules_local on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing unload_modules_local oleg328-server: modules unloaded. pdsh@oleg328-client: oleg328-client: ssh exited with exit code 2 pdsh@oleg328-client: oleg328-server: ssh exited with exit code 2 pdsh@oleg328-client: oleg328-client: ssh exited with exit code 2 pdsh@oleg328-client: oleg328-server: ssh exited with exit code 2