== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 08:54:22 (1787662462) start mds service on oleg403-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg403-server' oleg403-server: oleg403-server.virtnet: executing load_modules_local oleg403-server: Loading modules from /home/green/git/lustre-release/lustre oleg403-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg403-server: detected 4 online CPUs by sysfs oleg403-server: MODOPTS_LIBCFS= oleg403-server: Force libcfs to create 2 CPU partitions oleg403-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg403-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg403-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg403-server Start mds2: mount -t lustre -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg403-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-OST0000 oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg403-client.virtnet: -o user_xattr,flock 192.168.204.103@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 131 (time 1787662591.86 total 10.00 last 13.10) - open/close 342 (time 1787662601.87 total 20.02 last 21.07) - open/close 535 (time 1787662612.63 total 30.78 last 17.94) - open/close 691 (time 1787662623.71 total 41.86 last 14.08) - open/close 824 (time 1787662633.73 total 51.87 last 13.28) - open/close 1014 (time 1787662643.73 total 61.88 last 18.99) - open/close 1135 (time 1787662653.74 total 71.88 last 12.09) - open/close 1359 (time 1787662664.61 total 82.76 last 20.60) - open/close 1625 (time 1787662675.10 total 93.24 last 25.37) - open/close 1812 (time 1787662685.35 total 103.50 last 18.23) - open/close 2003 (time 1787662696.33 total 114.47 last 17.40) - open/close 2094 (time 1787662706.89 total 125.03 last 8.62) - open/close 2289 (time 1787662717.96 total 136.11 last 17.61) - open/close 2440 (time 1787662728.56 total 146.71 last 14.24) - open/close 2604 (time 1787662738.58 total 156.72 last 16.38) - open/close 2733 (time 1787662748.58 total 166.72 last 12.90) - open/close 2920 (time 1787662758.58 total 176.72 last 18.69) - open/close 3108 (time 1787662768.68 total 186.82 last 18.61) - open/close 3230 (time 1787662778.94 total 197.08 last 11.90) - open/close 3343 (time 1787662788.96 total 207.11 last 11.27) - open/close 3560 (time 1787662799.32 total 217.47 last 20.94) - open/close 3669 (time 1787662812.98 total 231.13 last 7.98) - open/close 3829 (time 1787662825.22 total 243.37 last 13.08) - open/close 4013 (time 1787662835.57 total 253.71 last 17.78) - open/close 4179 (time 1787662845.58 total 263.73 last 16.57) - open/close 4424 (time 1787662855.59 total 273.73 last 24.49) - open/close 4539 (time 1787662865.89 total 284.04 last 11.16) - open/close 4718 (time 1787662877.89 total 296.03 last 14.92) - open/close 4938 (time 1787662888.55 total 306.70 last 20.63) - open/close 5047 (time 1787662899.37 total 317.51 last 10.08) - open/close 5248 (time 1787662909.37 total 327.52 last 20.09) - open/close 5483 (time 1787662919.92 total 338.06 last 22.28) - open/close 5765 (time 1787662929.93 total 348.08 last 28.17) - open/close 5935 (time 1787662940.45 total 358.59 last 16.16) - open/close 6179 (time 1787662951.21 total 369.36 last 22.67) - open/close 6390 (time 1787662961.64 total 379.78 last 20.24) - open/close 6613 (time 1787662971.65 total 389.79 last 22.28) - open/close 6862 (time 1787662981.65 total 399.79 last 24.90) - open/close 7107 (time 1787662991.65 total 409.80 last 24.49) - open/close 7252 (time 1787663002.08 total 420.23 last 13.90) - open/close 7376 (time 1787663012.81 total 430.95 last 11.56) - open/close 7525 (time 1787663023.21 total 441.36 last 14.32) - open/close 7734 (time 1787663033.23 total 451.38 last 20.86) - open/close 7921 (time 1787663043.61 total 461.75 last 18.02) - open/close 8009 (time 1787663053.69 total 471.84 last 8.73) - open/close 8102 (time 1787663064.61 total 482.76 last 8.52) - open/close 8272 (time 1787663074.79 total 492.94 last 16.70) - open/close 8488 (time 1787663085.68 total 503.83 last 19.84) - open/close 8741 (time 1787663095.68 total 513.83 last 25.29) - open/close 8976 (time 1787663105.82 total 523.97 last 23.18) - open/close 9183 (time 1787663115.84 total 533.98 last 20.67) - open/close 9441 (time 1787663125.86 total 544.01 last 25.73) - open/close 9631 (time 1787663136.69 total 554.83 last 17.55) - open/close 9897 (time 1787663146.69 total 564.84 last 26.59) - open/close 9926 (time 1787663156.71 total 574.85 last 2.90) - open/close 10000 (time 1787663161.18 total 579.33 last 16.53) - open/close 10179 (time 1787663171.19 total 589.34 last 17.89) - open/close 10338 (time 1787663181.53 total 599.68 last 15.38) - open/close 10529 (time 1787663191.78 total 609.93 last 18.64) - open/close 10639 (time 1787663201.81 total 619.95 last 10.97) - open/close 10771 (time 1787663212.41 total 630.56 last 12.44) - open/close 10908 (time 1787663222.75 total 640.90 last 13.25) - open/close 11118 (time 1787663233.78 total 651.93 last 19.04) - open/close 11221 (time 1787663244.68 total 662.82 last 9.45) - open/close 11380 (time 1787663255.71 total 673.86 last 14.41) - open/close 11552 (time 1787663265.94 total 684.09 last 16.81) - open/close 11697 (time 1787663276.60 total 694.74 last 13.61) - open/close 11939 (time 1787663287.15 total 705.29 last 22.94) - open/close 12146 (time 1787663300.10 total 718.24 last 15.98) - open/close 12365 (time 1787663310.10 total 728.25 last 21.89) - open/close 12565 (time 1787663321.56 total 739.71 last 17.45) - open/close 12732 (time 1787663331.57 total 749.72 last 16.68) - open/close 12877 (time 1787663343.86 total 762.00 last 11.80) - open/close 13039 (time 1787663354.67 total 772.82 last 14.98) - open/close 13271 (time 1787663364.69 total 782.84 last 23.16) - open/close 13373 (time 1787663375.53 total 793.68 last 9.41) - open/close 13442 (time 1787663385.55 total 803.69 last 6.89) - open/close 13561 (time 1787663395.56 total 813.71 last 11.88) - open/close 13740 (time 1787663406.52 total 824.67 last 16.33) - open/close 13891 (time 1787663416.54 total 834.68 last 15.08) - open/close 14106 (time 1787663426.58 total 844.73 last 21.40) - open/close 14300 (time 1787663437.03 total 855.18 last 18.57) - open/close 14380 (time 1787663447.08 total 865.23 last 7.96) - open/close 14504 (time 1787663457.10 total 875.24 last 12.39) - open/close 14738 (time 1787663468.07 total 886.22 last 21.32) - open/close 14934 (time 1787663478.12 total 896.26 last 19.51) - open/close 15020 (time 1787663488.57 total 906.71 last 8.23) - open/close 15107 (time 1787663499.16 total 917.31 last 8.21) - open/close 15273 (time 1787663510.18 total 928.33 last 15.06) - open/close 15499 (time 1787663520.20 total 938.35 last 22.55) - open/close 15603 (time 1787663530.53 total 948.68 last 10.07) - open/close 15640 (time 1787663541.10 total 959.25 last 3.50) - open/close 15756 (time 1787663551.12 total 969.26 last 11.58) - open/close 15927 (time 1787663561.50 total 979.64 last 16.47) - open/close 15995 (time 1787663572.53 total 990.67 last 6.16) - open/close 16149 (time 1787663583.02 total 1001.17 last 14.68) - open/close 16393 (time 1787663593.04 total 1011.18 last 24.36) - open/close 16465 (time 1787663603.88 total 1022.03 last 6.64) - open/close 16541 (time 1787663613.95 total 1032.09 last 7.55) - open/close 16794 (time 1787663623.96 total 1042.11 last 25.27) - open/close 17392 (time 1787663633.99 total 1052.13 last 59.65) - open/close 17993 (time 1787663644.00 total 1062.14 last 60.03) - open/close 18547 (time 1787663654.00 total 1072.15 last 55.39) - open/close 19159 (time 1787663664.01 total 1082.15 last 61.17) - open/close 19692 (time 1787663674.01 total 1092.16 last 53.26) - open/close 20000 (time 1787663679.55 total 1097.69 last 55.68) - open/close 20543 (time 1787663689.55 total 1107.70 last 54.27) - open/close 21165 (time 1787663699.56 total 1117.70 last 62.15) - open/close 21806 (time 1787663709.58 total 1127.73 last 63.95) - open/close 22466 (time 1787663719.58 total 1137.73 last 66.00) - open/close 23092 (time 1787663729.58 total 1147.73 last 62.60) - open/close 23683 (time 1787663739.60 total 1157.75 last 58.98) - open/close 24325 (time 1787663749.62 total 1167.76 last 64.11) - open/close 25002 (time 1787663759.64 total 1177.78 last 67.57) - open/close 25531 (time 1787663769.64 total 1187.79 last 52.88) - open/close 25995 (time 1787663779.65 total 1197.79 last 46.37) - open/close 26622 (time 1787663789.67 total 1207.82 last 62.54) - open/close 27203 (time 1787663799.68 total 1217.83 last 58.06) - open/close 27825 (time 1787663809.68 total 1227.83 last 62.18) - open/close 28471 (time 1787663819.69 total 1237.84 last 64.57) - open/close 28982 (time 1787663829.70 total 1247.85 last 51.04) - open/close 29548 (time 1787663839.71 total 1257.86 last 56.53) - open/close 30000 (time 1787663848.65 total 1266.80 last 50.55) - open/close 30424 (time 1787663858.67 total 1276.82 last 42.31) - open/close 30902 (time 1787663868.68 total 1286.83 last 47.76) - open/close 31545 (time 1787663878.69 total 1296.83 last 64.27) - open/close 32181 (time 1787663888.69 total 1306.84 last 63.57) - open/close 32804 (time 1787663898.69 total 1316.84 last 62.28) - open/close 33372 (time 1787663908.70 total 1326.84 last 56.79) - open/close 33976 (time 1787663918.71 total 1336.85 last 60.34) - open/close 34606 (time 1787663928.73 total 1346.88 last 62.82) - open/close 35231 (time 1787663938.75 total 1356.89 last 62.41) - open/close 35874 (time 1787663948.76 total 1366.90 last 64.25) - open/close 36582 (time 1787663958.79 total 1376.93 last 70.60) - open/close 37190 (time 1787663968.79 total 1386.93 last 60.80) - open/close 37796 (time 1787663978.80 total 1396.95 last 60.51) - open/close 38387 (time 1787663988.81 total 1406.96 last 59.05) - open/close 39002 (time 1787663998.81 total 1416.96 last 61.48) - open/close 39627 (time 1787664008.82 total 1426.96 last 62.47) - open/close 40000 (time 1787664015.23 total 1433.37 last 58.18) - open/close 40655 (time 1787664025.23 total 1443.38 last 65.49) - open/close 41244 (time 1787664035.24 total 1453.39 last 58.84) - open/close 41775 (time 1787664045.25 total 1463.39 last 53.07) - open/close 42387 (time 1787664055.25 total 1473.40 last 61.16) - open/close 42940 (time 1787664065.25 total 1483.40 last 55.29) - open/close 43544 (time 1787664075.27 total 1493.42 last 60.31) - open/close 44156 (time 1787664085.28 total 1503.43 last 61.13) - open/close 44741 (time 1787664095.28 total 1513.43 last 58.48) - open/close 45262 (time 1787664105.29 total 1523.43 last 52.10) - open/close 45768 (time 1787664115.29 total 1533.44 last 50.58) - open/close 46330 (time 1787664125.30 total 1543.44 last 56.16) - open/close 47010 (time 1787664135.32 total 1553.46 last 67.88) - open/close 47668 (time 1787664145.32 total 1563.47 last 65.77) - open/close 48300 (time 1787664155.34 total 1573.48 last 63.10) - open/close 48883 (time 1787664165.34 total 1583.49 last 58.27) - open/close 49444 (time 1787664175.35 total 1593.49 last 56.06) total: 50000 open/close in 1602.04 seconds: 31.21 ops/second - unlinked 0 (time 1787664189 ; total 0 ; last 0) - unlinked 10000 (time 1787664372 ; total 183 ; last 183) - unlinked 20000 (time 1787664543 ; total 354 ; last 171) - unlinked 30000 (time 1787664722 ; total 533 ; last 179) - unlinked 40000 (time 1787664892 ; total 703 ; last 170) total: 50000 unlinks in 871 seconds: 57.405281 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg403-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg403-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg403-server unloading modules via unload_modules_local on: 'oleg403-server' oleg403-server: oleg403-server.virtnet: executing unload_modules_local oleg403-server: modules unloaded. pdsh@oleg403-client: oleg403-server: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-client: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-server: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-client: ssh exited with exit code 2