== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 21:40:55 (1788313255) start mds service on oleg243-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing load_modules_local oleg243-server: Loading modules from /home/green/git/lustre-release/lustre oleg243-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg243-server: detected 4 online CPUs by sysfs oleg243-server: MODOPTS_LIBCFS= oleg243-server: Force libcfs to create 2 CPU partitions oleg243-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg243-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg243-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg243-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock 192.168.202.143@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 265 (time 1788313344.29 total 10.02 last 26.46) - open/close 441 (time 1788313354.67 total 20.40 last 16.95) - open/close 597 (time 1788313364.90 total 30.62 last 15.26) - open/close 736 (time 1788313375.02 total 40.75 last 13.73) - open/close 786 (time 1788313385.85 total 51.58 last 4.62) - open/close 943 (time 1788313396.92 total 62.64 last 14.19) - open/close 1016 (time 1788313406.94 total 72.66 last 7.29) - open/close 1129 (time 1788313416.97 total 82.69 last 11.26) - open/close 1312 (time 1788313427.23 total 92.95 last 17.83) - open/close 1495 (time 1788313437.24 total 102.97 last 18.28) - open/close 1716 (time 1788313448.28 total 114.00 last 20.02) - open/close 1944 (time 1788313458.47 total 124.19 last 22.38) - open/close 2098 (time 1788313468.76 total 134.49 last 14.95) - open/close 2311 (time 1788313478.78 total 144.51 last 21.26) - open/close 2551 (time 1788313488.79 total 154.51 last 23.99) - open/close 2746 (time 1788313498.91 total 164.63 last 19.26) - open/close 2942 (time 1788313510.13 total 175.85 last 17.48) - open/close 3172 (time 1788313520.13 total 185.85 last 23.00) - open/close 3344 (time 1788313530.53 total 196.26 last 16.53) - open/close 3540 (time 1788313541.25 total 206.97 last 18.29) - open/close 3736 (time 1788313551.34 total 217.07 last 19.42) - open/close 3885 (time 1788313561.35 total 227.07 last 14.89) - open/close 4017 (time 1788313571.69 total 237.42 last 12.76) - open/close 4240 (time 1788313582.51 total 248.24 last 20.61) - open/close 4413 (time 1788313593.50 total 259.22 last 15.75) - open/close 4592 (time 1788313603.96 total 269.68 last 17.12) - open/close 4739 (time 1788313614.34 total 280.06 last 14.16) - open/close 4903 (time 1788313624.67 total 290.39 last 15.87) - open/close 5099 (time 1788313634.69 total 300.42 last 19.55) - open/close 5286 (time 1788313646.76 total 312.49 last 15.49) - open/close 5490 (time 1788313657.15 total 322.87 last 19.65) - open/close 5726 (time 1788313667.63 total 333.35 last 22.52) - open/close 5958 (time 1788313678.05 total 343.78 last 22.25) - open/close 6130 (time 1788313688.07 total 353.79 last 17.18) - open/close 6352 (time 1788313698.34 total 364.06 last 21.61) - open/close 6480 (time 1788313710.36 total 376.09 last 10.65) - open/close 6708 (time 1788313720.94 total 386.66 last 21.55) - open/close 6964 (time 1788313730.94 total 396.66 last 25.60) - open/close 7185 (time 1788313741.23 total 406.95 last 21.48) - open/close 7435 (time 1788313751.24 total 416.97 last 24.97) - open/close 7527 (time 1788313761.33 total 427.05 last 9.12) - open/close 7661 (time 1788313771.89 total 437.62 last 12.69) - open/close 7772 (time 1788313781.93 total 447.65 last 11.06) - open/close 7926 (time 1788313792.49 total 458.21 last 14.59) - open/close 8144 (time 1788313803.20 total 468.93 last 20.34) - open/close 8299 (time 1788313813.22 total 478.94 last 15.48) - open/close 8488 (time 1788313823.68 total 489.41 last 18.06) - open/close 8670 (time 1788313833.69 total 499.41 last 18.20) - open/close 8941 (time 1788313843.70 total 509.42 last 27.06) - open/close 9154 (time 1788313853.94 total 519.66 last 20.80) - open/close 9289 (time 1788313866.28 total 532.00 last 10.94) - open/close 9500 (time 1788313876.28 total 542.01 last 21.10) - open/close 9749 (time 1788313887.06 total 552.79 last 23.09) - open/close 10000 (time 1788313895.84 total 561.56 last 28.61) - open/close 10228 (time 1788313906.56 total 572.28 last 21.27) - open/close 10436 (time 1788313916.56 total 582.28 last 20.79) - open/close 10642 (time 1788313926.58 total 592.30 last 20.57) - open/close 10814 (time 1788313936.58 total 602.31 last 17.19) - open/close 11068 (time 1788313946.58 total 612.31 last 25.40) - open/close 11193 (time 1788313957.22 total 622.95 last 11.75) - open/close 11418 (time 1788313968.05 total 633.78 last 20.78) - open/close 11653 (time 1788313978.06 total 643.78 last 23.48) - open/close 11879 (time 1788313988.75 total 654.47 last 21.14) - open/close 12078 (time 1788314000.01 total 665.73 last 17.67) - open/close 12252 (time 1788314010.86 total 676.59 last 16.03) - open/close 12494 (time 1788314020.87 total 686.60 last 24.17) - open/close 12734 (time 1788314030.87 total 696.60 last 24.00) - open/close 12990 (time 1788314040.88 total 706.60 last 25.59) - open/close 13222 (time 1788314051.42 total 717.14 last 22.01) - open/close 13426 (time 1788314061.47 total 727.19 last 20.30) - open/close 13629 (time 1788314071.48 total 737.20 last 20.28) - open/close 13860 (time 1788314082.26 total 747.99 last 21.42) - open/close 14022 (time 1788314092.29 total 758.01 last 16.16) - open/close 14207 (time 1788314102.31 total 768.03 last 18.47) - open/close 14405 (time 1788314113.19 total 778.92 last 18.19) - open/close 14599 (time 1788314123.21 total 788.94 last 19.36) - open/close 14813 (time 1788314133.22 total 798.95 last 21.38) - open/close 15024 (time 1788314143.26 total 808.98 last 21.02) - open/close 15122 (time 1788314154.29 total 820.01 last 8.88) - open/close 15288 (time 1788314164.85 total 830.57 last 15.72) - open/close 15513 (time 1788314175.72 total 841.44 last 20.70) - open/close 15635 (time 1788314185.76 total 851.49 last 12.14) - open/close 15795 (time 1788314195.82 total 861.54 last 15.92) - open/close 15958 (time 1788314206.30 total 872.03 last 15.54) - open/close 16127 (time 1788314216.68 total 882.41 last 16.28) - open/close 16360 (time 1788314226.69 total 892.41 last 23.29) - open/close 16545 (time 1788314236.71 total 902.44 last 18.46) - open/close 16662 (time 1788314246.95 total 912.67 last 11.43) - open/close 16797 (time 1788314256.96 total 922.69 last 13.48) - open/close 16951 (time 1788314267.84 total 933.56 last 14.16) - open/close 17169 (time 1788314278.25 total 943.97 last 20.94) - open/close 17473 (time 1788314290.99 total 956.71 last 23.87) - open/close 17653 (time 1788314301.48 total 967.21 last 17.15) - open/close 17823 (time 1788314311.50 total 977.22 last 16.98) - open/close 18048 (time 1788314322.16 total 987.89 last 21.10) - open/close 18098 (time 1788314333.02 total 998.75 last 4.60) - open/close 18662 (time 1788314343.03 total 1008.75 last 56.37) - open/close 19363 (time 1788314353.03 total 1018.76 last 70.05) - open/close 20000 (time 1788314362.66 total 1028.38 last 66.19) - open/close 20601 (time 1788314372.67 total 1038.39 last 60.03) - open/close 21287 (time 1788314382.67 total 1048.40 last 68.57) - open/close 21922 (time 1788314392.68 total 1058.40 last 63.46) - open/close 22500 (time 1788314402.69 total 1068.41 last 57.75) - open/close 23150 (time 1788314412.71 total 1078.43 last 64.86) - open/close 23753 (time 1788314422.74 total 1088.46 last 60.12) - open/close 24307 (time 1788314432.75 total 1098.47 last 55.36) - open/close 24949 (time 1788314442.75 total 1108.47 last 64.18) - open/close 25563 (time 1788314452.75 total 1118.48 last 61.39) - open/close 26204 (time 1788314462.76 total 1128.48 last 64.08) - open/close 26901 (time 1788314472.76 total 1138.48 last 69.68) - open/close 27672 (time 1788314482.76 total 1148.49 last 77.06) - open/close 28232 (time 1788314492.76 total 1158.49 last 55.99) - open/close 28899 (time 1788314502.78 total 1168.50 last 66.62) - open/close 29536 (time 1788314512.78 total 1178.50 last 63.68) - open/close 30000 (time 1788314520.26 total 1185.99 last 62.00) - open/close 30677 (time 1788314530.32 total 1196.04 last 67.32) - open/close 31297 (time 1788314540.33 total 1206.05 last 61.96) - open/close 31951 (time 1788314550.33 total 1216.06 last 65.35) - open/close 32662 (time 1788314560.34 total 1226.07 last 71.04) - open/close 33420 (time 1788314570.35 total 1236.08 last 75.71) - open/close 34146 (time 1788314580.36 total 1246.09 last 72.53) - open/close 34740 (time 1788314590.37 total 1256.09 last 59.37) - open/close 35409 (time 1788314600.37 total 1266.10 last 66.89) - open/close 36120 (time 1788314610.39 total 1276.12 last 70.94) - open/close 36723 (time 1788314620.40 total 1286.13 last 60.24) - open/close 37396 (time 1788314630.43 total 1296.15 last 67.14) - open/close 37987 (time 1788314640.43 total 1306.16 last 59.07) - open/close 38536 (time 1788314650.45 total 1316.17 last 54.81) - open/close 38980 (time 1788314660.46 total 1326.19 last 44.35) - open/close 39637 (time 1788314670.46 total 1336.19 last 65.68) - open/close 40000 (time 1788314675.03 total 1340.75 last 79.52) - open/close 40725 (time 1788314685.03 total 1350.76 last 72.48) - open/close 41409 (time 1788314695.05 total 1360.77 last 68.31) - open/close 42154 (time 1788314705.05 total 1370.77 last 74.48) - open/close 42898 (time 1788314715.06 total 1380.79 last 74.29) - open/close 43619 (time 1788314725.08 total 1390.80 last 72.01) - open/close 44285 (time 1788314735.12 total 1400.84 last 66.34) - open/close 44947 (time 1788314745.13 total 1410.85 last 66.13) - open/close 45632 (time 1788314755.13 total 1420.86 last 68.46) - open/close 46306 (time 1788314765.15 total 1430.87 last 67.31) - open/close 46925 (time 1788314775.15 total 1440.87 last 61.89) - open/close 47643 (time 1788314785.15 total 1450.88 last 71.77) - open/close 48309 (time 1788314795.15 total 1460.88 last 66.59) - open/close 48955 (time 1788314805.16 total 1470.89 last 64.55) - open/close 49607 (time 1788314815.16 total 1480.89 last 65.19) total: 50000 open/close in 1486.34 seconds: 33.64 ops/second - unlinked 0 (time 1788314825 ; total 0 ; last 0) - unlinked 10000 (time 1788314971 ; total 146 ; last 146) - unlinked 20000 (time 1788315131 ; total 306 ; last 160) - unlinked 30000 (time 1788315294 ; total 469 ; last 163) - unlinked 40000 (time 1788315449 ; total 624 ; last 155) total: 50000 unlinks in 804 seconds: 62.189056 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server unloading modules via unload_modules_local on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing unload_modules_local oleg243-server: modules unloaded. pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2