== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 07:37:05 (1787398625) start mds service on oleg103-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg103-server' oleg103-server: oleg103-server.virtnet: executing load_modules_local oleg103-server: Loading modules from /home/green/git/lustre-release/lustre oleg103-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg103-server: detected 4 online CPUs by sysfs oleg103-server: MODOPTS_LIBCFS= oleg103-server: Force libcfs to create 2 CPU partitions oleg103-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg103-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg103-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg103-server: oleg103-server.virtnet: executing set_default_debug -1 all pdsh@oleg103-client: oleg103-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg103-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg103-server: oleg103-server.virtnet: executing set_default_debug -1 all pdsh@oleg103-client: oleg103-server: ssh exited with exit code 1 Started lustre-OST0000 oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg103-client.virtnet: -o user_xattr,flock 192.168.201.103@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 119 (time 1787398717.34 total 10.80 last 11.01) - open/close 262 (time 1787398727.72 total 21.18 last 13.78) - open/close 372 (time 1787398737.72 total 31.18 last 11.00) - open/close 531 (time 1787398747.82 total 41.28 last 15.74) - open/close 632 (time 1787398757.85 total 51.31 last 10.07) - open/close 765 (time 1787398768.56 total 62.03 last 12.41) - open/close 844 (time 1787398778.98 total 72.44 last 7.58) - open/close 905 (time 1787398788.98 total 82.45 last 6.10) - open/close 1015 (time 1787398799.47 total 92.94 last 10.49) - open/close 1232 (time 1787398809.48 total 102.94 last 21.69) - open/close 1390 (time 1787398820.50 total 113.96 last 14.34) - open/close 1579 (time 1787398830.68 total 124.14 last 18.56) - open/close 1726 (time 1787398841.17 total 134.64 last 14.01) - open/close 1894 (time 1787398851.68 total 145.15 last 15.98) - open/close 2001 (time 1787398861.69 total 155.15 last 10.70) - open/close 2125 (time 1787398872.55 total 166.01 last 11.42) - open/close 2212 (time 1787398883.03 total 176.50 last 8.30) - open/close 2381 (time 1787398893.04 total 186.51 last 16.88) - open/close 2574 (time 1787398903.33 total 196.79 last 18.77) - open/close 2703 (time 1787398913.72 total 207.18 last 12.41) - open/close 2814 (time 1787398924.68 total 218.15 last 10.13) - open/close 2902 (time 1787398935.28 total 228.74 last 8.30) - open/close 3006 (time 1787398945.29 total 238.75 last 10.39) - open/close 3174 (time 1787398955.50 total 248.96 last 16.46) - open/close 3388 (time 1787398965.50 total 258.96 last 21.40) - open/close 3570 (time 1787398975.50 total 268.97 last 18.19) - open/close 3738 (time 1787398985.95 total 279.41 last 16.08) - open/close 3865 (time 1787398995.97 total 289.43 last 12.67) - open/close 4108 (time 1787399005.98 total 299.44 last 24.28) - open/close 4266 (time 1787399016.48 total 309.94 last 15.05) - open/close 4362 (time 1787399027.83 total 321.29 last 8.46) - open/close 4431 (time 1787399038.92 total 332.39 last 6.22) - open/close 4578 (time 1787399048.93 total 342.39 last 14.69) - open/close 4675 (time 1787399058.99 total 352.45 last 9.64) - open/close 4785 (time 1787399069.73 total 363.19 last 10.24) - open/close 4979 (time 1787399079.74 total 373.21 last 19.37) - open/close 5090 (time 1787399089.97 total 383.43 last 10.85) - open/close 5211 (time 1787399100.95 total 394.41 last 11.02) - open/close 5308 (time 1787399112.02 total 405.49 last 8.76) - open/close 5562 (time 1787399123.12 total 416.58 last 22.89) - open/close 5748 (time 1787399133.67 total 427.14 last 17.62) - open/close 5897 (time 1787399144.93 total 438.39 last 13.24) - open/close 6084 (time 1787399154.94 total 448.41 last 18.67) - open/close 6235 (time 1787399165.42 total 458.89 last 14.41) - open/close 6376 (time 1787399176.23 total 469.69 last 13.05) - open/close 6403 (time 1787399186.39 total 479.86 last 2.66) - open/close 6485 (time 1787399199.14 total 492.61 last 6.43) - open/close 6618 (time 1787399209.15 total 502.61 last 13.29) - open/close 6838 (time 1787399219.15 total 512.62 last 21.99) - open/close 6970 (time 1787399231.92 total 525.38 last 10.34) - open/close 7139 (time 1787399241.94 total 535.40 last 16.88) - open/close 7365 (time 1787399251.95 total 545.41 last 22.58) - open/close 7595 (time 1787399261.96 total 555.42 last 22.97) - open/close 7824 (time 1787399271.96 total 565.43 last 22.90) - open/close 8012 (time 1787399281.98 total 575.44 last 18.77) - open/close 8170 (time 1787399292.19 total 585.66 last 15.47) - open/close 8344 (time 1787399302.77 total 596.23 last 16.45) - open/close 8600 (time 1787399313.45 total 606.91 last 23.98) - open/close 8760 (time 1787399323.46 total 616.92 last 15.98) - open/close 8911 (time 1787399333.46 total 626.93 last 15.10) - open/close 9121 (time 1787399343.85 total 637.31 last 20.22) - open/close 9330 (time 1787399354.11 total 647.57 last 20.37) - open/close 9623 (time 1787399364.82 total 658.29 last 27.34) - open/close 9831 (time 1787399374.83 total 668.29 last 20.79) - open/close 10000 (time 1787399384.47 total 677.93 last 17.53) - open/close 10232 (time 1787399394.59 total 688.06 last 22.92) - open/close 10484 (time 1787399405.41 total 698.88 last 23.29) - open/close 10738 (time 1787399415.42 total 708.88 last 25.39) - open/close 10983 (time 1787399426.34 total 719.81 last 22.42) - open/close 11200 (time 1787399436.35 total 729.81 last 21.70) - open/close 11384 (time 1787399447.33 total 740.79 last 16.75) - open/close 11636 (time 1787399457.43 total 750.89 last 24.95) - open/close 11892 (time 1787399467.44 total 760.90 last 25.58) - open/close 12111 (time 1787399478.37 total 771.83 last 20.04) - open/close 12259 (time 1787399488.46 total 781.92 last 14.66) - open/close 12405 (time 1787399498.53 total 791.99 last 14.50) - open/close 12580 (time 1787399508.55 total 802.01 last 17.47) - open/close 12712 (time 1787399518.96 total 812.42 last 12.68) - open/close 12901 (time 1787399529.32 total 822.78 last 18.25) - open/close 13069 (time 1787399539.32 total 832.78 last 16.80) - open/close 13279 (time 1787399549.60 total 843.06 last 20.43) - open/close 13523 (time 1787399560.27 total 853.73 last 22.86) - open/close 13712 (time 1787399570.29 total 863.75 last 18.87) - open/close 13861 (time 1787399580.41 total 873.87 last 14.72) - open/close 13939 (time 1787399590.62 total 884.08 last 7.64) - open/close 14195 (time 1787399601.62 total 895.08 last 23.28) - open/close 14392 (time 1787399612.53 total 905.99 last 18.05) - open/close 14634 (time 1787399623.01 total 916.47 last 23.10) - open/close 14712 (time 1787399633.04 total 926.50 last 7.78) - open/close 14979 (time 1787399643.04 total 936.51 last 26.69) - open/close 15166 (time 1787399654.07 total 947.53 last 16.96) - open/close 15379 (time 1787399665.10 total 958.57 last 19.31) - open/close 15516 (time 1787399675.11 total 968.57 last 13.69) - open/close 15639 (time 1787399685.11 total 978.57 last 12.30) - open/close 15848 (time 1787399695.12 total 988.58 last 20.89) - open/close 16007 (time 1787399706.03 total 999.49 last 14.57) - open/close 16161 (time 1787399716.05 total 1009.52 last 15.37) - open/close 16398 (time 1787399726.74 total 1020.21 last 22.17) - open/close 16636 (time 1787399736.75 total 1030.21 last 23.78) - open/close 17191 (time 1787399746.76 total 1040.22 last 55.48) - open/close 17810 (time 1787399756.76 total 1050.22 last 61.87) - open/close 18368 (time 1787399766.76 total 1060.22 last 55.79) - open/close 19038 (time 1787399776.77 total 1070.23 last 66.97) - open/close 19699 (time 1787399786.80 total 1080.26 last 65.91) - open/close 20000 (time 1787399791.02 total 1084.48 last 71.26) - open/close 20750 (time 1787399801.02 total 1094.49 last 74.96) - open/close 21512 (time 1787399811.03 total 1104.49 last 76.18) - open/close 22240 (time 1787399821.04 total 1114.50 last 72.73) - open/close 22900 (time 1787399831.04 total 1124.50 last 66.00) - open/close 23537 (time 1787399841.05 total 1134.51 last 63.61) - open/close 24306 (time 1787399851.06 total 1144.52 last 76.84) - open/close 24968 (time 1787399861.06 total 1154.52 last 66.20) - open/close 25583 (time 1787399871.06 total 1164.53 last 61.48) - open/close 26159 (time 1787399881.07 total 1174.54 last 57.54) - open/close 26840 (time 1787399891.08 total 1184.54 last 68.08) - open/close 27435 (time 1787399901.08 total 1194.54 last 59.48) - open/close 28096 (time 1787399911.09 total 1204.55 last 66.02) - open/close 28590 (time 1787399921.10 total 1214.56 last 49.38) - open/close 29318 (time 1787399931.10 total 1224.56 last 72.78) - open/close 29993 (time 1787399941.11 total 1234.57 last 67.44) - open/close 30644 (time 1787399951.12 total 1244.58 last 65.03) - open/close 31321 (time 1787399961.12 total 1254.58 last 67.68) - open/close 32083 (time 1787399971.13 total 1264.59 last 76.15) - open/close 32834 (time 1787399981.13 total 1274.59 last 75.07) - open/close 33568 (time 1787399991.13 total 1284.60 last 73.38) - open/close 34244 (time 1787400001.14 total 1294.60 last 67.55) - open/close 34934 (time 1787400011.14 total 1304.61 last 68.99) - open/close 35607 (time 1787400021.16 total 1314.62 last 67.21) - open/close 36350 (time 1787400031.17 total 1324.63 last 74.23) - open/close 37009 (time 1787400041.17 total 1334.64 last 65.85) - open/close 37634 (time 1787400051.17 total 1344.64 last 62.49) - open/close 38290 (time 1787400061.19 total 1354.65 last 65.52) - open/close 38830 (time 1787400071.24 total 1364.70 last 53.73) - open/close 39459 (time 1787400081.25 total 1374.72 last 62.79) - open/close 39937 (time 1787400091.27 total 1384.73 last 47.75) - open/close 40456 (time 1787400101.27 total 1394.73 last 51.89) - open/close 41054 (time 1787400111.27 total 1404.74 last 59.77) - open/close 41669 (time 1787400121.28 total 1414.74 last 61.47) - open/close 42378 (time 1787400131.28 total 1424.74 last 70.89) - open/close 43104 (time 1787400141.29 total 1434.75 last 72.55) - open/close 43813 (time 1787400151.29 total 1444.75 last 70.90) - open/close 44406 (time 1787400161.31 total 1454.78 last 59.16) - open/close 45021 (time 1787400171.32 total 1464.78 last 61.49) - open/close 45651 (time 1787400181.32 total 1474.78 last 62.99) - open/close 46383 (time 1787400191.33 total 1484.79 last 73.14) - open/close 47102 (time 1787400201.33 total 1494.79 last 71.87) - open/close 47756 (time 1787400211.35 total 1504.81 last 65.27) - open/close 48479 (time 1787400221.35 total 1514.82 last 72.27) - open/close 49179 (time 1787400231.36 total 1524.83 last 69.94) - open/close 49902 (time 1787400241.36 total 1534.83 last 72.29) total: 50000 open/close in 1536.10 seconds: 32.55 ops/second - unlinked 0 (time 1787400247 ; total 0 ; last 0) - unlinked 10000 (time 1787400405 ; total 158 ; last 158) - unlinked 20000 (time 1787400566 ; total 319 ; last 161) - unlinked 30000 (time 1787400721 ; total 474 ; last 155) - unlinked 40000 (time 1787400896 ; total 649 ; last 175) total: 50000 unlinks in 810 seconds: 61.728394 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg103-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg103-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg103-server stop mds service on oleg103-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg103-server unloading modules via unload_modules_local on: 'oleg103-server' oleg103-server: oleg103-server.virtnet: executing unload_modules_local oleg103-server: modules unloaded. pdsh@oleg103-client: oleg103-server: ssh exited with exit code 2 pdsh@oleg103-client: oleg103-client: ssh exited with exit code 2 pdsh@oleg103-client: oleg103-server: ssh exited with exit code 2 pdsh@oleg103-client: oleg103-client: ssh exited with exit code 2