== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:59:31 (1788443971) start mds service on oleg442-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg442-server' oleg442-server: oleg442-server.virtnet: executing load_modules_local oleg442-server: Loading modules from /home/green/git/lustre-release/lustre oleg442-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg442-server: detected 4 online CPUs by sysfs oleg442-server: MODOPTS_LIBCFS= oleg442-server: Force libcfs to create 2 CPU partitions oleg442-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg442-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg442-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg442-server: oleg442-server.virtnet: executing set_default_debug -1 all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg442-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg442-server: oleg442-server.virtnet: executing set_default_debug -1 all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-OST0000 oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg442-client.virtnet: -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 146 (time 1788444065.90 total 12.51 last 11.67) - open/close 238 (time 1788444075.92 total 22.54 last 9.18) - open/close 338 (time 1788444088.76 total 35.38 last 7.79) - open/close 408 (time 1788444099.01 total 45.63 last 6.83) - open/close 468 (time 1788444109.37 total 55.98 last 5.79) - open/close 574 (time 1788444119.40 total 66.02 last 10.57) - open/close 640 (time 1788444130.55 total 77.16 last 5.92) - open/close 821 (time 1788444140.57 total 87.18 last 18.06) - open/close 965 (time 1788444151.00 total 97.62 last 13.80) - open/close 1118 (time 1788444161.72 total 108.33 last 14.28) - open/close 1257 (time 1788444171.98 total 118.59 last 13.55) - open/close 1448 (time 1788444183.51 total 130.13 last 16.56) - open/close 1537 (time 1788444194.81 total 141.42 last 7.88) - open/close 1621 (time 1788444205.02 total 151.64 last 8.22) - open/close 1747 (time 1788444215.03 total 161.65 last 12.59) - open/close 1921 (time 1788444225.05 total 171.66 last 17.37) - open/close 2109 (time 1788444235.05 total 181.67 last 18.79) - open/close 2311 (time 1788444245.41 total 192.02 last 19.51) - open/close 2477 (time 1788444255.42 total 202.03 last 16.58) - open/close 2639 (time 1788444265.50 total 212.12 last 16.07) - open/close 2846 (time 1788444278.05 total 224.67 last 16.50) - open/close 3071 (time 1788444288.08 total 234.69 last 22.44) - open/close 3240 (time 1788444299.01 total 245.62 last 15.46) - open/close 3413 (time 1788444309.16 total 255.77 last 17.04) - open/close 3557 (time 1788444319.86 total 266.47 last 13.46) - open/close 3648 (time 1788444332.84 total 279.46 last 7.01) - open/close 3794 (time 1788444343.79 total 290.41 last 13.33) - open/close 3896 (time 1788444354.61 total 301.22 last 9.43) - open/close 4047 (time 1788444365.60 total 312.22 last 13.74) - open/close 4230 (time 1788444376.68 total 323.30 last 16.51) - open/close 4360 (time 1788444386.68 total 333.30 last 13.00) - open/close 4412 (time 1788444397.67 total 344.29 last 4.73) - open/close 4594 (time 1788444407.69 total 354.30 last 18.18) - open/close 4727 (time 1788444418.10 total 364.71 last 12.78) - open/close 4806 (time 1788444428.38 total 375.00 last 7.68) - open/close 4835 (time 1788444438.38 total 385.00 last 2.90) - open/close 4992 (time 1788444449.43 total 396.05 last 14.21) - open/close 5109 (time 1788444460.14 total 406.76 last 10.93) - open/close 5264 (time 1788444470.16 total 416.77 last 15.48) - open/close 5447 (time 1788444480.16 total 426.78 last 18.29) - open/close 5559 (time 1788444490.39 total 437.01 last 10.94) - open/close 5632 (time 1788444500.42 total 447.04 last 7.28) - open/close 5782 (time 1788444511.22 total 457.84 last 13.89) - open/close 5842 (time 1788444522.15 total 468.76 last 5.49) - open/close 5991 (time 1788444533.73 total 480.35 last 12.86) - open/close 6165 (time 1788444543.73 total 490.35 last 17.40) - open/close 6349 (time 1788444553.74 total 500.36 last 18.38) - open/close 6550 (time 1788444563.77 total 510.38 last 20.05) - open/close 6680 (time 1788444573.91 total 520.53 last 12.82) - open/close 6886 (time 1788444586.86 total 533.47 last 15.91) - open/close 7099 (time 1788444597.80 total 544.42 last 19.46) - open/close 7302 (time 1788444607.83 total 554.44 last 20.25) - open/close 7521 (time 1788444618.47 total 565.08 last 20.58) - open/close 7745 (time 1788444628.47 total 575.09 last 22.39) - open/close 7877 (time 1788444639.56 total 586.18 last 11.90) - open/close 8066 (time 1788444649.96 total 596.58 last 18.17) - open/close 8175 (time 1788444660.01 total 606.62 last 10.85) - open/close 8251 (time 1788444670.95 total 617.57 last 6.95) - open/close 8418 (time 1788444681.38 total 627.99 last 16.02) - open/close 8564 (time 1788444691.39 total 638.00 last 14.59) - open/close 8719 (time 1788444702.23 total 648.84 last 14.30) - open/close 8904 (time 1788444712.25 total 658.87 last 18.45) - open/close 9085 (time 1788444722.26 total 668.88 last 18.09) - open/close 9336 (time 1788444732.28 total 678.89 last 25.06) - open/close 9580 (time 1788444742.37 total 688.99 last 24.16) - open/close 9755 (time 1788444752.38 total 698.99 last 17.50) - open/close 9963 (time 1788444763.11 total 709.73 last 19.37) - open/close 10000 (time 1788444768.04 total 714.66 last 7.51) - open/close 10151 (time 1788444778.46 total 725.08 last 14.49) - open/close 10361 (time 1788444789.11 total 735.72 last 19.72) - open/close 10537 (time 1788444799.92 total 746.54 last 16.28) - open/close 10706 (time 1788444809.94 total 756.55 last 16.87) - open/close 10906 (time 1788444820.72 total 767.33 last 18.56) - open/close 11074 (time 1788444830.76 total 777.38 last 16.73) - open/close 11162 (time 1788444840.78 total 787.39 last 8.78) - open/close 11326 (time 1788444851.75 total 798.36 last 14.95) - open/close 11446 (time 1788444862.70 total 809.31 last 10.96) - open/close 11564 (time 1788444872.80 total 819.42 last 11.68) - open/close 11732 (time 1788444882.82 total 829.43 last 16.77) - open/close 11864 (time 1788444893.47 total 840.08 last 12.39) - open/close 11979 (time 1788444903.48 total 850.09 last 11.49) - open/close 12141 (time 1788444913.83 total 860.44 last 15.65) - open/close 12268 (time 1788444924.34 total 870.96 last 12.08) - open/close 12354 (time 1788444934.35 total 880.96 last 8.60) - open/close 12519 (time 1788444945.17 total 891.79 last 15.24) - open/close 12685 (time 1788444955.20 total 901.81 last 16.56) - open/close 12781 (time 1788444965.29 total 911.91 last 9.51) - open/close 12878 (time 1788444975.85 total 922.47 last 9.18) - open/close 12994 (time 1788444987.42 total 934.04 last 10.03) - open/close 13182 (time 1788444997.44 total 944.06 last 18.77) - open/close 13298 (time 1788445012.26 total 958.87 last 7.83) - open/close 13397 (time 1788445023.36 total 969.98 last 8.91) - open/close 13508 (time 1788445034.35 total 980.96 last 10.11) - open/close 13627 (time 1788445045.11 total 991.73 last 11.05) - open/close 13677 (time 1788445055.93 total 1002.55 last 4.62) - open/close 13748 (time 1788445065.93 total 1012.55 last 7.10) - open/close 13916 (time 1788445076.84 total 1023.46 last 15.40) - open/close 13990 (time 1788445087.78 total 1034.40 last 6.76) - open/close 14147 (time 1788445098.59 total 1045.20 last 14.53) - open/close 14309 (time 1788445109.28 total 1055.89 last 15.15) - open/close 14389 (time 1788445120.48 total 1067.10 last 7.14) - open/close 14519 (time 1788445130.72 total 1077.33 last 12.70) - open/close 14714 (time 1788445140.90 total 1087.51 last 19.15) - open/close 14864 (time 1788445151.71 total 1098.33 last 13.87) - open/close 15052 (time 1788445162.46 total 1109.08 last 17.49) - open/close 15204 (time 1788445173.15 total 1119.76 last 14.22) - open/close 15477 (time 1788445183.16 total 1129.78 last 27.26) - open/close 16069 (time 1788445193.17 total 1139.78 last 59.18) - open/close 16595 (time 1788445203.17 total 1149.79 last 52.58) - open/close 17178 (time 1788445213.18 total 1159.80 last 58.24) - open/close 17704 (time 1788445223.25 total 1169.87 last 52.24) - open/close 18157 (time 1788445233.25 total 1179.87 last 45.29) - open/close 18682 (time 1788445243.26 total 1189.87 last 52.49) - open/close 19291 (time 1788445253.26 total 1199.87 last 60.89) - open/close 19925 (time 1788445263.27 total 1209.88 last 63.34) - open/close 20553 (time 1788445273.32 total 1219.94 last 62.46) - open/close 21136 (time 1788445283.34 total 1229.95 last 58.20) - open/close 21713 (time 1788445293.34 total 1239.96 last 57.67) - open/close 22303 (time 1788445303.36 total 1249.97 last 58.91) - open/close 22904 (time 1788445313.37 total 1259.98 last 60.03) - open/close 23480 (time 1788445323.37 total 1269.99 last 57.57) - open/close 24021 (time 1788445333.38 total 1280.00 last 54.05) - open/close 24537 (time 1788445343.41 total 1290.02 last 51.48) - open/close 25050 (time 1788445353.42 total 1300.03 last 51.24) - open/close 25593 (time 1788445363.43 total 1310.04 last 54.27) - open/close 26195 (time 1788445373.43 total 1320.04 last 60.18) - open/close 26760 (time 1788445383.44 total 1330.05 last 56.45) - open/close 27272 (time 1788445393.44 total 1340.06 last 51.19) - open/close 27840 (time 1788445403.44 total 1350.06 last 56.78) - open/close 28458 (time 1788445413.48 total 1360.09 last 61.58) - open/close 29160 (time 1788445423.49 total 1370.10 last 70.15) - open/close 29757 (time 1788445433.49 total 1380.11 last 59.66) - open/close 30000 (time 1788445437.82 total 1384.43 last 56.19) - open/close 30453 (time 1788445447.87 total 1394.49 last 45.04) - open/close 30927 (time 1788445457.88 total 1404.50 last 47.37) - open/close 31358 (time 1788445467.90 total 1414.51 last 43.03) - open/close 31821 (time 1788445477.91 total 1424.52 last 46.25) - open/close 32440 (time 1788445487.92 total 1434.53 last 61.84) - open/close 32982 (time 1788445497.94 total 1444.56 last 54.07) - open/close 33605 (time 1788445507.94 total 1454.56 last 62.29) - open/close 34164 (time 1788445517.95 total 1464.57 last 55.87) - open/close 34765 (time 1788445527.97 total 1474.58 last 60.01) - open/close 35371 (time 1788445537.98 total 1484.59 last 60.53) - open/close 36002 (time 1788445547.98 total 1494.60 last 63.06) - open/close 36563 (time 1788445558.00 total 1504.61 last 56.03) - open/close 37102 (time 1788445568.00 total 1514.62 last 53.86) - open/close 37643 (time 1788445578.02 total 1524.63 last 54.03) - open/close 38207 (time 1788445588.02 total 1534.63 last 56.39) - open/close 38794 (time 1788445598.05 total 1544.67 last 58.50) - open/close 39417 (time 1788445608.07 total 1554.69 last 62.19) - open/close 39976 (time 1788445618.07 total 1564.69 last 55.90) - open/close 40486 (time 1788445628.09 total 1574.71 last 50.88) - open/close 40992 (time 1788445638.11 total 1584.72 last 50.54) - open/close 41534 (time 1788445648.11 total 1594.73 last 54.17) - open/close 42078 (time 1788445658.13 total 1604.75 last 54.31) - open/close 42678 (time 1788445668.13 total 1614.75 last 59.98) - open/close 43226 (time 1788445678.14 total 1624.75 last 54.78) - open/close 43767 (time 1788445688.15 total 1634.76 last 54.04) - open/close 44491 (time 1788445698.16 total 1644.77 last 72.33) - open/close 45110 (time 1788445708.17 total 1654.79 last 61.82) - open/close 45660 (time 1788445718.18 total 1664.80 last 54.95) - open/close 46180 (time 1788445728.18 total 1674.80 last 51.98) - open/close 46788 (time 1788445738.20 total 1684.82 last 60.69) - open/close 47415 (time 1788445748.21 total 1694.82 last 62.66) - open/close 48068 (time 1788445758.21 total 1704.83 last 65.27) - open/close 48688 (time 1788445768.23 total 1714.84 last 61.90) - open/close 49300 (time 1788445778.24 total 1724.85 last 61.14) - open/close 49960 (time 1788445788.26 total 1734.87 last 65.86) total: 50000 open/close in 1735.53 seconds: 28.81 ops/second - unlinked 0 (time 1788445793 ; total 0 ; last 0) - unlinked 10000 (time 1788445977 ; total 184 ; last 184) - unlinked 20000 (time 1788446157 ; total 364 ; last 180) - unlinked 30000 (time 1788446324 ; total 531 ; last 167) - unlinked 40000 (time 1788446484 ; total 691 ; last 160) total: 50000 unlinks in 854 seconds: 58.548008 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg442-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg442-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg442-server stop mds service on oleg442-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg442-server unloading modules via unload_modules_local on: 'oleg442-server' oleg442-server: oleg442-server.virtnet: executing unload_modules_local oleg442-server: modules unloaded. pdsh@oleg442-client: oleg442-server: ssh exited with exit code 2 pdsh@oleg442-client: oleg442-client: ssh exited with exit code 2 pdsh@oleg442-client: oleg442-server: ssh exited with exit code 2 pdsh@oleg442-client: oleg442-client: ssh exited with exit code 2