== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 14:50:01 (1780339801) start mds service on oleg444-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg444-server' oleg444-server: oleg444-server.virtnet: executing load_modules_local oleg444-server: Loading modules from /home/green/git/lustre-release/lustre oleg444-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg444-server: detected 4 online CPUs by sysfs oleg444-server: MODOPTS_LIBCFS= oleg444-server: Force libcfs to create 2 CPU partitions oleg444-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg444-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg444-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg444-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-OST0000 oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg444-client.virtnet: -o user_xattr,flock 192.168.204.144@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 164 (time 1780339877.48 total 10.46 last 15.68) - open/close 267 (time 1780339887.48 total 20.46 last 10.30) - open/close 419 (time 1780339897.49 total 30.47 last 15.19) - open/close 634 (time 1780339907.50 total 40.48 last 21.48) - open/close 752 (time 1780339917.54 total 50.52 last 11.76) - open/close 919 (time 1780339927.79 total 60.77 last 16.29) - open/close 1056 (time 1780339939.80 total 72.78 last 11.41) - open/close 1236 (time 1780339950.04 total 83.02 last 17.57) - open/close 1395 (time 1780339960.21 total 93.19 last 15.64) - open/close 1538 (time 1780339970.60 total 103.58 last 13.76) - open/close 1787 (time 1780339980.63 total 113.61 last 24.82) - open/close 1971 (time 1780339990.65 total 123.63 last 18.38) - open/close 2186 (time 1780340000.66 total 133.64 last 21.46) - open/close 2374 (time 1780340011.17 total 144.15 last 17.89) - open/close 2506 (time 1780340021.71 total 154.69 last 12.52) - open/close 2685 (time 1780340031.72 total 164.70 last 17.88) - open/close 2857 (time 1780340042.21 total 175.19 last 16.41) - open/close 3018 (time 1780340052.22 total 185.20 last 16.08) - open/close 3220 (time 1780340063.00 total 195.98 last 18.74) - open/close 3413 (time 1780340073.01 total 205.98 last 19.29) - open/close 3540 (time 1780340083.29 total 216.27 last 12.34) - open/close 3759 (time 1780340093.32 total 226.30 last 21.84) - open/close 3875 (time 1780340104.21 total 237.19 last 10.65) - open/close 4063 (time 1780340114.34 total 247.32 last 18.57) - open/close 4178 (time 1780340124.95 total 257.93 last 10.83) - open/close 4349 (time 1780340135.01 total 267.99 last 16.99) - open/close 4480 (time 1780340145.04 total 278.01 last 13.07) - open/close 4629 (time 1780340155.16 total 288.14 last 14.71) - open/close 4750 (time 1780340165.55 total 298.53 last 11.65) - open/close 4826 (time 1780340175.56 total 308.54 last 7.59) - open/close 4985 (time 1780340186.39 total 319.37 last 14.67) - open/close 5144 (time 1780340196.41 total 329.39 last 15.87) - open/close 5277 (time 1780340207.14 total 340.12 last 12.40) - open/close 5357 (time 1780340217.16 total 350.14 last 7.98) - open/close 5495 (time 1780340227.33 total 360.31 last 13.58) - open/close 5605 (time 1780340238.59 total 371.57 last 9.76) - open/close 5822 (time 1780340249.07 total 382.05 last 20.72) - open/close 5997 (time 1780340259.18 total 392.16 last 17.31) - open/close 6074 (time 1780340270.37 total 403.35 last 6.88) - open/close 6230 (time 1780340281.24 total 414.22 last 14.35) - open/close 6422 (time 1780340293.87 total 426.85 last 15.20) - open/close 6584 (time 1780340303.88 total 436.86 last 16.19) - open/close 6724 (time 1780340314.02 total 447.00 last 13.80) - open/close 6805 (time 1780340324.12 total 457.10 last 8.02) - open/close 6931 (time 1780340334.56 total 467.54 last 12.07) - open/close 7027 (time 1780340345.32 total 478.30 last 8.92) - open/close 7147 (time 1780340355.37 total 488.35 last 11.93) - open/close 7206 (time 1780340367.26 total 500.24 last 4.96) - open/close 7333 (time 1780340377.28 total 510.26 last 12.68) - open/close 7433 (time 1780340387.88 total 520.86 last 9.43) - open/close 7604 (time 1780340397.98 total 530.96 last 16.93) - open/close 7758 (time 1780340407.99 total 540.97 last 15.39) - open/close 7862 (time 1780340418.64 total 551.62 last 9.76) - open/close 7946 (time 1780340428.67 total 561.65 last 8.38) - open/close 8104 (time 1780340442.48 total 575.46 last 11.44) - open/close 8283 (time 1780340452.52 total 585.50 last 17.84) - open/close 8446 (time 1780340462.53 total 595.51 last 16.28) - open/close 8516 (time 1780340472.73 total 605.71 last 6.86) - open/close 8700 (time 1780340483.37 total 616.35 last 17.30) - open/close 8896 (time 1780340498.02 total 631.00 last 13.38) - open/close 9008 (time 1780340508.03 total 641.01 last 11.19) - open/close 9123 (time 1780340518.04 total 651.02 last 11.49) - open/close 9342 (time 1780340528.99 total 661.97 last 19.99) - open/close 9496 (time 1780340539.55 total 672.53 last 14.59) - open/close 9674 (time 1780340549.95 total 682.93 last 17.12) - open/close 9909 (time 1780340560.85 total 693.83 last 21.55) - open/close 10000 (time 1780340569.73 total 702.71 last 10.25) - open/close 10123 (time 1780340579.74 total 712.72 last 12.29) - open/close 10292 (time 1780340589.75 total 722.73 last 16.88) - open/close 10361 (time 1780340600.35 total 733.33 last 6.51) - open/close 10441 (time 1780340610.36 total 743.34 last 7.99) - open/close 10626 (time 1780340620.37 total 753.35 last 18.49) - open/close 10749 (time 1780340631.07 total 764.05 last 11.49) - open/close 10872 (time 1780340641.08 total 774.06 last 12.29) - open/close 11035 (time 1780340651.09 total 784.07 last 16.28) - open/close 11241 (time 1780340662.82 total 795.80 last 17.56) - open/close 11356 (time 1780340673.64 total 806.62 last 10.63) - open/close 11397 (time 1780340683.93 total 816.91 last 3.99) - open/close 11639 (time 1780340693.93 total 826.91 last 24.20) - open/close 11717 (time 1780340704.63 total 837.61 last 7.29) - open/close 11779 (time 1780340719.97 total 852.95 last 4.04) - open/close 12015 (time 1780340730.60 total 863.57 last 22.22) - open/close 12213 (time 1780340740.61 total 873.59 last 19.77) - open/close 12437 (time 1780340751.57 total 884.55 last 20.44) - open/close 12560 (time 1780340761.57 total 894.55 last 12.29) - open/close 12634 (time 1780340773.10 total 906.08 last 6.42) - open/close 12714 (time 1780340783.40 total 916.38 last 7.76) - open/close 12833 (time 1780340793.69 total 926.67 last 11.56) - open/close 12953 (time 1780340803.71 total 936.69 last 11.98) - open/close 13081 (time 1780340813.98 total 946.96 last 12.47) - open/close 13284 (time 1780340824.35 total 957.33 last 19.57) - open/close 13518 (time 1780340834.37 total 967.35 last 23.37) - open/close 13751 (time 1780340845.35 total 978.33 last 21.21) - open/close 13836 (time 1780340855.98 total 988.95 last 8.00) - open/close 13928 (time 1780340866.73 total 999.71 last 8.56) - open/close 14036 (time 1780340877.61 total 1010.59 last 9.93) - open/close 14260 (time 1780340888.66 total 1021.64 last 20.26) - open/close 14306 (time 1780340899.31 total 1032.29 last 4.32) - open/close 14417 (time 1780340910.30 total 1043.28 last 10.09) - open/close 14495 (time 1780340920.87 total 1053.85 last 7.38) - open/close 14626 (time 1780340931.86 total 1064.83 last 11.92) - open/close 14729 (time 1780340943.04 total 1076.02 last 9.21) - open/close 15278 (time 1780340953.04 total 1086.02 last 54.88) - open/close 15818 (time 1780340963.06 total 1096.04 last 53.89) - open/close 16406 (time 1780340973.06 total 1106.04 last 58.78) - open/close 16918 (time 1780340983.07 total 1116.05 last 51.17) - open/close 17398 (time 1780340993.09 total 1126.07 last 47.91) - open/close 17920 (time 1780341003.10 total 1136.08 last 52.12) - open/close 18412 (time 1780341013.16 total 1146.14 last 48.90) - open/close 18927 (time 1780341023.18 total 1156.16 last 51.40) - open/close 19476 (time 1780341033.19 total 1166.17 last 54.86) - open/close 19965 (time 1780341043.20 total 1176.18 last 48.88) - open/close 20465 (time 1780341053.20 total 1186.18 last 49.97) - open/close 20982 (time 1780341063.23 total 1196.21 last 51.58) - open/close 21472 (time 1780341073.24 total 1206.22 last 48.93) - open/close 22004 (time 1780341083.25 total 1216.23 last 53.14) - open/close 22418 (time 1780341093.27 total 1226.25 last 41.35) - open/close 22769 (time 1780341103.27 total 1236.25 last 35.08) - open/close 23198 (time 1780341113.29 total 1246.27 last 42.81) - open/close 23707 (time 1780341123.31 total 1256.28 last 50.85) - open/close 24232 (time 1780341133.31 total 1266.29 last 52.48) - open/close 24687 (time 1780341143.32 total 1276.30 last 45.46) - open/close 25189 (time 1780341153.34 total 1286.32 last 50.10) - open/close 25615 (time 1780341163.34 total 1296.32 last 42.57) - open/close 26121 (time 1780341173.35 total 1306.33 last 50.58) - open/close 26517 (time 1780341183.36 total 1316.34 last 39.56) - open/close 26955 (time 1780341193.37 total 1326.35 last 43.75) - open/close 27520 (time 1780341203.37 total 1336.35 last 56.47) - open/close 28059 (time 1780341213.38 total 1346.36 last 53.89) - open/close 28537 (time 1780341223.38 total 1356.36 last 47.78) - open/close 29105 (time 1780341233.38 total 1366.36 last 56.79) - open/close 29609 (time 1780341243.39 total 1376.37 last 50.36) - open/close 30000 (time 1780341250.05 total 1383.03 last 58.76) - open/close 30512 (time 1780341260.07 total 1393.05 last 51.06) - open/close 31072 (time 1780341270.09 total 1403.07 last 55.90) - open/close 31581 (time 1780341280.10 total 1413.08 last 50.86) - open/close 32146 (time 1780341290.10 total 1423.08 last 56.48) - open/close 32734 (time 1780341300.12 total 1433.10 last 58.71) - open/close 33281 (time 1780341310.13 total 1443.11 last 54.64) - open/close 33876 (time 1780341320.13 total 1453.11 last 59.47) - open/close 34423 (time 1780341330.15 total 1463.13 last 54.62) - open/close 34987 (time 1780341340.15 total 1473.13 last 56.36) - open/close 35508 (time 1780341350.16 total 1483.14 last 52.05) - open/close 36084 (time 1780341360.19 total 1493.17 last 57.42) - open/close 36651 (time 1780341370.20 total 1503.18 last 56.65) - open/close 37189 (time 1780341380.22 total 1513.20 last 53.70) - open/close 37758 (time 1780341390.23 total 1523.21 last 56.85) - open/close 38329 (time 1780341400.25 total 1533.23 last 56.99) - open/close 38922 (time 1780341410.28 total 1543.26 last 59.14) - open/close 39450 (time 1780341420.29 total 1553.27 last 52.74) - open/close 39995 (time 1780341430.29 total 1563.27 last 54.50) - open/close 40512 (time 1780341440.31 total 1573.29 last 51.62) - open/close 41092 (time 1780341450.32 total 1583.30 last 57.93) - open/close 41616 (time 1780341460.33 total 1593.31 last 52.34) - open/close 42089 (time 1780341470.34 total 1603.32 last 47.26) - open/close 42588 (time 1780341480.35 total 1613.33 last 49.85) - open/close 43088 (time 1780341490.37 total 1623.35 last 49.90) - open/close 43594 (time 1780341500.39 total 1633.37 last 50.50) - open/close 44125 (time 1780341510.39 total 1643.37 last 53.09) - open/close 44712 (time 1780341520.40 total 1653.38 last 58.65) - open/close 45239 (time 1780341530.41 total 1663.39 last 52.64) - open/close 45786 (time 1780341540.42 total 1673.40 last 54.63) - open/close 46365 (time 1780341550.43 total 1683.41 last 57.85) - open/close 46895 (time 1780341560.46 total 1693.44 last 52.84) - open/close 47449 (time 1780341570.47 total 1703.45 last 55.37) - open/close 48025 (time 1780341580.47 total 1713.45 last 57.58) - open/close 48588 (time 1780341590.49 total 1723.47 last 56.22) - open/close 49168 (time 1780341600.50 total 1733.48 last 57.91) - open/close 49715 (time 1780341610.52 total 1743.50 last 54.61) total: 50000 open/close in 1748.48 seconds: 28.60 ops/second - unlinked 0 (time 1780341619 ; total 0 ; last 0) - unlinked 10000 (time 1780341786 ; total 167 ; last 167) - unlinked 20000 (time 1780341920 ; total 301 ; last 134) - unlinked 30000 (time 1780342042 ; total 423 ; last 122) - unlinked 40000 (time 1780342164 ; total 545 ; last 122) total: 50000 unlinks in 666 seconds: 75.075073 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg444-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg444-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg444-server stop mds service on oleg444-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg444-server unloading modules via unload_modules_local on: 'oleg444-server' oleg444-server: oleg444-server.virtnet: executing unload_modules_local oleg444-server: modules unloaded. pdsh@oleg444-client: oleg444-client: ssh exited with exit code 2 pdsh@oleg444-client: oleg444-server: ssh exited with exit code 2 pdsh@oleg444-client: oleg444-server: ssh exited with exit code 2 pdsh@oleg444-client: oleg444-client: ssh exited with exit code 2