== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:02:55 (1783137775) start mds service on oleg226-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg226-server' oleg226-server: oleg226-server.virtnet: executing load_modules_local oleg226-server: Loading modules from /home/green/git/lustre-release/lustre oleg226-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg226-server: detected 4 online CPUs by sysfs oleg226-server: MODOPTS_LIBCFS= oleg226-server: Force libcfs to create 2 CPU partitions oleg226-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg226-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg226-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg226-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-OST0000 oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg226-client.virtnet: -o user_xattr,flock 192.168.202.126@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 202 (time 1783137868.32 total 10.63 last 19.00) - open/close 414 (time 1783137878.39 total 20.70 last 21.05) - open/close 638 (time 1783137889.07 total 31.39 last 20.97) - open/close 887 (time 1783137899.11 total 41.42 last 24.82) - open/close 1153 (time 1783137909.15 total 51.46 last 26.50) - open/close 1400 (time 1783137920.09 total 62.41 last 22.56) - open/close 1606 (time 1783137932.98 total 75.30 last 15.98) - open/close 1766 (time 1783137943.01 total 85.32 last 15.95) - open/close 1863 (time 1783137955.43 total 97.74 last 7.81) - open/close 2113 (time 1783137967.80 total 110.11 last 20.21) - open/close 2357 (time 1783137978.45 total 120.76 last 22.91) - open/close 2550 (time 1783137988.49 total 130.80 last 19.22) - open/close 2779 (time 1783137998.77 total 141.08 last 22.28) - open/close 3002 (time 1783138008.80 total 151.11 last 22.22) - open/close 3212 (time 1783138019.37 total 161.68 last 19.87) - open/close 3402 (time 1783138029.49 total 171.80 last 18.78) - open/close 3641 (time 1783138039.50 total 181.81 last 23.88) - open/close 3812 (time 1783138050.80 total 193.11 last 15.13) - open/close 3978 (time 1783138060.80 total 203.11 last 16.59) - open/close 4111 (time 1783138070.89 total 213.20 last 13.19) - open/close 4302 (time 1783138081.51 total 223.82 last 17.98) - open/close 4544 (time 1783138091.52 total 233.83 last 24.18) - open/close 4737 (time 1783138101.81 total 244.12 last 18.76) - open/close 4832 (time 1783138112.74 total 255.06 last 8.69) - open/close 4932 (time 1783138123.12 total 265.44 last 9.63) - open/close 5165 (time 1783138133.87 total 276.18 last 21.69) - open/close 5309 (time 1783138143.88 total 286.19 last 14.39) - open/close 5550 (time 1783138153.89 total 296.20 last 24.08) - open/close 5744 (time 1783138164.26 total 306.57 last 18.70) - open/close 5975 (time 1783138174.91 total 317.22 last 21.70) - open/close 6183 (time 1783138185.65 total 327.96 last 19.37) - open/close 6263 (time 1783138196.13 total 338.44 last 7.63) - open/close 6428 (time 1783138206.27 total 348.59 last 16.27) - open/close 6508 (time 1783138216.28 total 358.60 last 7.99) - open/close 6703 (time 1783138227.07 total 369.38 last 18.08) - open/close 6920 (time 1783138237.11 total 379.42 last 21.62) - open/close 7118 (time 1783138247.11 total 389.42 last 19.79) - open/close 7198 (time 1783138257.92 total 400.23 last 7.40) - open/close 7438 (time 1783138267.95 total 410.26 last 23.94) - open/close 7682 (time 1783138278.88 total 421.19 last 22.31) - open/close 7818 (time 1783138289.00 total 431.31 last 13.44) - open/close 7885 (time 1783138299.00 total 441.32 last 6.70) - open/close 8083 (time 1783138309.88 total 452.20 last 18.20) - open/close 8240 (time 1783138320.81 total 463.12 last 14.37) - open/close 8384 (time 1783138330.83 total 473.14 last 14.37) - open/close 8489 (time 1783138340.85 total 483.16 last 10.48) - open/close 8712 (time 1783138350.85 total 493.16 last 22.30) - open/close 8835 (time 1783138361.52 total 503.84 last 11.52) - open/close 9009 (time 1783138372.04 total 514.35 last 16.55) - open/close 9194 (time 1783138382.08 total 524.39 last 18.42) - open/close 9355 (time 1783138392.49 total 534.80 last 15.47) - open/close 9594 (time 1783138402.49 total 544.80 last 23.89) - open/close 9807 (time 1783138413.56 total 555.87 last 19.24) - open/close 10000 (time 1783138419.27 total 561.58 last 33.80) - open/close 10277 (time 1783138429.41 total 571.73 last 27.31) - open/close 10462 (time 1783138440.23 total 582.54 last 17.11) - open/close 10693 (time 1783138450.23 total 592.55 last 23.08) - open/close 10720 (time 1783138461.10 total 603.41 last 2.49) - open/close 10933 (time 1783138471.96 total 614.28 last 19.61) - open/close 11089 (time 1783138481.97 total 624.28 last 15.59) - open/close 11259 (time 1783138491.98 total 634.29 last 16.98) - open/close 11390 (time 1783138502.00 total 644.31 last 13.08) - open/close 11526 (time 1783138512.37 total 654.68 last 13.12) - open/close 11688 (time 1783138527.16 total 669.48 last 10.95) - open/close 11736 (time 1783138538.80 total 681.11 last 4.13) - open/close 11837 (time 1783138550.06 total 692.37 last 8.97) - open/close 11905 (time 1783138560.75 total 703.06 last 6.36) - open/close 12134 (time 1783138571.36 total 713.68 last 21.58) - open/close 12386 (time 1783138581.37 total 723.69 last 25.17) - open/close 12520 (time 1783138591.97 total 734.28 last 12.65) - open/close 12644 (time 1783138602.22 total 744.53 last 12.10) - open/close 12765 (time 1783138612.32 total 754.63 last 11.98) - open/close 12836 (time 1783138622.35 total 764.66 last 7.08) - open/close 13065 (time 1783138633.27 total 775.58 last 20.97) - open/close 13274 (time 1783138643.27 total 785.58 last 20.89) - open/close 13482 (time 1783138653.69 total 796.00 last 19.96) - open/close 13733 (time 1783138663.70 total 806.01 last 25.08) - open/close 13892 (time 1783138673.72 total 816.03 last 15.87) - open/close 14018 (time 1783138685.48 total 827.80 last 10.71) - open/close 14282 (time 1783138695.67 total 837.98 last 25.92) - open/close 14531 (time 1783138705.68 total 847.99 last 24.87) - open/close 14770 (time 1783138716.65 total 858.96 last 21.79) - open/close 15011 (time 1783138726.66 total 868.97 last 24.06) - open/close 15204 (time 1783138737.14 total 879.45 last 18.42) - open/close 15395 (time 1783138747.14 total 889.45 last 19.09) - open/close 15521 (time 1783138757.25 total 899.56 last 12.47) - open/close 15598 (time 1783138767.55 total 909.86 last 7.48) - open/close 15733 (time 1783138777.55 total 919.86 last 13.50) - open/close 15926 (time 1783138787.56 total 929.88 last 19.27) - open/close 16217 (time 1783138797.57 total 939.88 last 29.09) - open/close 16427 (time 1783138808.49 total 950.80 last 19.23) - open/close 16684 (time 1783138819.29 total 961.60 last 23.79) - open/close 16971 (time 1783138829.29 total 971.61 last 28.69) - open/close 17210 (time 1783138839.59 total 981.90 last 23.22) - open/close 17433 (time 1783138849.89 total 992.20 last 21.65) - open/close 17698 (time 1783138859.90 total 1002.21 last 26.46) - open/close 17964 (time 1783138869.92 total 1012.23 last 26.56) - open/close 18186 (time 1783138879.93 total 1022.24 last 22.18) - open/close 19053 (time 1783138889.93 total 1032.24 last 86.65) - open/close 19956 (time 1783138899.93 total 1042.24 last 90.30) - open/close 20688 (time 1783138909.95 total 1052.26 last 73.09) - open/close 21491 (time 1783138919.95 total 1062.26 last 80.27) - open/close 22292 (time 1783138929.96 total 1072.27 last 80.04) - open/close 23103 (time 1783138939.97 total 1082.28 last 81.04) - open/close 23999 (time 1783138949.97 total 1092.28 last 89.59) - open/close 24772 (time 1783138959.97 total 1102.29 last 77.25) - open/close 25576 (time 1783138969.99 total 1112.30 last 80.26) - open/close 26156 (time 1783138979.99 total 1122.30 last 58.00) - open/close 26951 (time 1783138990.00 total 1132.31 last 79.45) - open/close 27854 (time 1783139000.00 total 1142.32 last 90.24) - open/close 28741 (time 1783139010.00 total 1152.32 last 88.70) - open/close 29565 (time 1783139020.00 total 1162.32 last 82.39) - open/close 30000 (time 1783139025.77 total 1168.08 last 75.43) - open/close 30774 (time 1783139035.80 total 1178.11 last 77.21) - open/close 31388 (time 1783139045.80 total 1188.11 last 61.38) - open/close 32049 (time 1783139055.81 total 1198.12 last 66.06) - open/close 32692 (time 1783139065.81 total 1208.12 last 64.26) - open/close 33495 (time 1783139075.81 total 1218.12 last 80.30) - open/close 34344 (time 1783139085.82 total 1228.13 last 84.86) - open/close 35240 (time 1783139095.82 total 1238.14 last 89.55) - open/close 36059 (time 1783139105.83 total 1248.14 last 81.84) - open/close 36806 (time 1783139115.87 total 1258.18 last 74.44) - open/close 37666 (time 1783139125.88 total 1268.19 last 85.89) - open/close 38446 (time 1783139135.90 total 1278.21 last 77.84) - open/close 39323 (time 1783139145.90 total 1288.22 last 87.65) - open/close 40000 (time 1783139153.34 total 1295.65 last 91.05) - open/close 40803 (time 1783139163.35 total 1305.66 last 80.25) - open/close 41538 (time 1783139173.35 total 1315.66 last 73.46) - open/close 42246 (time 1783139183.36 total 1325.68 last 70.72) - open/close 43077 (time 1783139193.37 total 1335.69 last 83.02) - open/close 43891 (time 1783139203.38 total 1345.70 last 81.32) - open/close 44641 (time 1783139213.39 total 1355.70 last 74.96) - open/close 45405 (time 1783139223.39 total 1365.71 last 76.36) - open/close 46229 (time 1783139233.40 total 1375.71 last 82.35) - open/close 47083 (time 1783139243.40 total 1385.71 last 85.38) - open/close 47857 (time 1783139253.42 total 1395.73 last 77.26) - open/close 48608 (time 1783139263.45 total 1405.76 last 74.90) - open/close 49389 (time 1783139273.45 total 1415.77 last 78.04) total: 50000 open/close in 1424.87 seconds: 35.09 ops/second - unlinked 0 (time 1783139287 ; total 0 ; last 0) - unlinked 10000 (time 1783139420 ; total 133 ; last 133) - unlinked 20000 (time 1783139557 ; total 270 ; last 137) - unlinked 30000 (time 1783139691 ; total 404 ; last 134) - unlinked 40000 (time 1783139831 ; total 544 ; last 140) total: 50000 unlinks in 682 seconds: 73.313782 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg226-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg226-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg226-server stop mds service on oleg226-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg226-server unloading modules via unload_modules_local on: 'oleg226-server' oleg226-server: oleg226-server.virtnet: executing unload_modules_local oleg226-server: modules unloaded. pdsh@oleg226-client: oleg226-client: ssh exited with exit code 2 pdsh@oleg226-client: oleg226-server: ssh exited with exit code 2 pdsh@oleg226-client: oleg226-client: ssh exited with exit code 2 pdsh@oleg226-client: oleg226-server: ssh exited with exit code 2