== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:34:26 (1788600866) start mds service on oleg334-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg334-server' oleg334-server: oleg334-server.virtnet: executing load_modules_local oleg334-server: Loading modules from /home/green/git/lustre-release/lustre oleg334-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg334-server: detected 4 online CPUs by sysfs oleg334-server: MODOPTS_LIBCFS= oleg334-server: Force libcfs to create 2 CPU partitions oleg334-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg334-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg334-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg334-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-OST0000 oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg334-client.virtnet: -o user_xattr,flock 192.168.203.134@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 216 (time 1788600936.33 total 10.01 last 21.58) - open/close 446 (time 1788600946.34 total 20.01 last 22.98) - open/close 669 (time 1788600956.34 total 30.02 last 22.29) - open/close 954 (time 1788600967.12 total 40.79 last 26.45) - open/close 1165 (time 1788600977.14 total 50.82 last 21.05) - open/close 1382 (time 1788600987.84 total 61.51 last 20.29) - open/close 1602 (time 1788600998.86 total 72.54 last 19.95) - open/close 1827 (time 1788601009.47 total 83.15 last 21.20) - open/close 2047 (time 1788601020.29 total 93.96 last 20.34) - open/close 2272 (time 1788601030.31 total 103.98 last 22.46) - open/close 2494 (time 1788601040.32 total 113.99 last 22.18) - open/close 2736 (time 1788601051.08 total 124.76 last 22.49) - open/close 2923 (time 1788601061.09 total 134.76 last 18.69) - open/close 3041 (time 1788601071.12 total 144.79 last 11.76) - open/close 3177 (time 1788601082.34 total 156.02 last 12.11) - open/close 3246 (time 1788601092.75 total 166.43 last 6.63) - open/close 3469 (time 1788601103.68 total 177.35 last 20.41) - open/close 3621 (time 1788601113.68 total 187.36 last 15.19) - open/close 3817 (time 1788601123.69 total 197.36 last 19.59) - open/close 4048 (time 1788601133.71 total 207.38 last 23.06) - open/close 4272 (time 1788601144.01 total 217.69 last 21.74) - open/close 4493 (time 1788601154.07 total 227.75 last 21.96) - open/close 4744 (time 1788601164.08 total 237.76 last 25.08) - open/close 4975 (time 1788601174.79 total 248.47 last 21.57) - open/close 5158 (time 1788601185.62 total 259.29 last 16.90) - open/close 5355 (time 1788601196.55 total 270.23 last 18.02) - open/close 5509 (time 1788601206.69 total 280.37 last 15.18) - open/close 5671 (time 1788601218.38 total 292.06 last 13.86) - open/close 5893 (time 1788601229.25 total 302.93 last 20.42) - open/close 6097 (time 1788601239.91 total 313.58 last 19.15) - open/close 6331 (time 1788601249.92 total 323.60 last 23.37) - open/close 6549 (time 1788601259.93 total 333.61 last 21.77) - open/close 6763 (time 1788601269.95 total 343.62 last 21.37) - open/close 6900 (time 1788601280.48 total 354.15 last 13.01) - open/close 7117 (time 1788601290.65 total 364.33 last 21.33) - open/close 7250 (time 1788601300.75 total 374.43 last 13.17) - open/close 7431 (time 1788601310.94 total 384.62 last 17.76) - open/close 7636 (time 1788601322.13 total 395.81 last 18.32) - open/close 7865 (time 1788601332.13 total 405.81 last 22.89) - open/close 8023 (time 1788601344.96 total 418.64 last 12.31) - open/close 8191 (time 1788601354.99 total 428.67 last 16.76) - open/close 8321 (time 1788601365.46 total 439.14 last 12.41) - open/close 8538 (time 1788601375.94 total 449.61 last 20.72) - open/close 8769 (time 1788601386.43 total 460.10 last 22.02) - open/close 8929 (time 1788601396.43 total 470.11 last 15.99) - open/close 9162 (time 1788601407.07 total 480.75 last 21.90) - open/close 9365 (time 1788601417.08 total 490.75 last 20.29) - open/close 9560 (time 1788601428.30 total 501.98 last 17.37) - open/close 9790 (time 1788601438.67 total 512.34 last 22.18) - open/close 9973 (time 1788601449.68 total 523.35 last 16.62) - open/close 10158 (time 1788601460.23 total 533.91 last 17.52) - open/close 10339 (time 1788601472.64 total 546.32 last 14.59) - open/close 10541 (time 1788601483.60 total 557.28 last 18.43) - open/close 10765 (time 1788601493.60 total 567.28 last 22.39) - open/close 10997 (time 1788601504.40 total 578.08 last 21.49) - open/close 11217 (time 1788601515.10 total 588.77 last 20.56) - open/close 11441 (time 1788601525.14 total 598.81 last 22.31) - open/close 11661 (time 1788601535.83 total 609.51 last 20.57) - open/close 11844 (time 1788601546.42 total 620.10 last 17.28) - open/close 12049 (time 1788601556.43 total 630.11 last 20.47) - open/close 12320 (time 1788601566.43 total 640.11 last 27.10) - open/close 12524 (time 1788601576.64 total 650.32 last 19.99) - open/close 12767 (time 1788601586.64 total 660.32 last 24.29) - open/close 13004 (time 1788601597.59 total 671.26 last 21.66) - open/close 13224 (time 1788601607.97 total 681.64 last 21.20) - open/close 13429 (time 1788601617.97 total 691.65 last 20.49) - open/close 13652 (time 1788601628.00 total 701.68 last 22.24) - open/close 13844 (time 1788601638.08 total 711.76 last 19.04) - open/close 14051 (time 1788601648.09 total 721.77 last 20.69) - open/close 14278 (time 1788601659.03 total 732.70 last 20.75) - open/close 14497 (time 1788601669.46 total 743.13 last 21.00) - open/close 14697 (time 1788601679.47 total 753.15 last 19.96) - open/close 14919 (time 1788601689.49 total 763.17 last 22.16) - open/close 15119 (time 1788601700.20 total 773.87 last 18.68) - open/close 15305 (time 1788601710.69 total 784.37 last 17.72) - open/close 15434 (time 1788601720.70 total 794.38 last 12.89) - open/close 15618 (time 1788601730.90 total 804.58 last 18.04) - open/close 15781 (time 1788601742.14 total 815.82 last 14.51) - open/close 16006 (time 1788601752.15 total 825.83 last 22.47) - open/close 16231 (time 1788601762.16 total 835.84 last 22.48) - open/close 16426 (time 1788601772.20 total 845.87 last 19.43) - open/close 16648 (time 1788601782.21 total 855.88 last 22.18) - open/close 16877 (time 1788601792.22 total 865.90 last 22.87) - open/close 17072 (time 1788601802.50 total 876.18 last 18.97) - open/close 17265 (time 1788601812.51 total 886.18 last 19.29) - open/close 17520 (time 1788601823.39 total 897.07 last 23.42) - open/close 17741 (time 1788601833.59 total 907.27 last 21.66) - open/close 17973 (time 1788601843.84 total 917.52 last 22.65) - open/close 18228 (time 1788601854.55 total 928.22 last 23.82) - open/close 18751 (time 1788601864.55 total 938.23 last 52.26) - open/close 19437 (time 1788601874.57 total 948.24 last 68.50) - open/close 20000 (time 1788601883.08 total 956.76 last 66.11) - open/close 20566 (time 1788601893.08 total 966.76 last 56.59) - open/close 21171 (time 1788601903.11 total 976.79 last 60.35) - open/close 21728 (time 1788601913.12 total 986.80 last 55.65) - open/close 22386 (time 1788601923.13 total 996.81 last 65.74) - open/close 23080 (time 1788601933.13 total 1006.81 last 69.38) - open/close 23768 (time 1788601943.14 total 1016.82 last 68.70) - open/close 24443 (time 1788601953.15 total 1026.83 last 67.47) - open/close 25110 (time 1788601963.16 total 1036.84 last 66.61) - open/close 25770 (time 1788601973.18 total 1046.85 last 65.92) - open/close 26435 (time 1788601983.19 total 1056.86 last 66.43) - open/close 27075 (time 1788601993.20 total 1066.87 last 63.95) - open/close 27715 (time 1788602003.20 total 1076.88 last 63.97) - open/close 28287 (time 1788602013.21 total 1086.89 last 57.14) - open/close 28764 (time 1788602023.22 total 1096.89 last 47.67) - open/close 29254 (time 1788602033.24 total 1106.91 last 48.90) - open/close 29784 (time 1788602043.24 total 1116.92 last 52.96) - open/close 30000 (time 1788602046.96 total 1120.64 last 58.13) - open/close 30622 (time 1788602056.96 total 1130.64 last 62.19) - open/close 31270 (time 1788602066.96 total 1140.64 last 64.79) - open/close 31911 (time 1788602076.97 total 1150.65 last 64.02) - open/close 32587 (time 1788602087.00 total 1160.68 last 67.44) - open/close 33258 (time 1788602097.03 total 1170.71 last 66.89) - open/close 33834 (time 1788602107.05 total 1180.73 last 57.48) - open/close 34470 (time 1788602117.05 total 1190.73 last 63.58) - open/close 35064 (time 1788602127.06 total 1200.74 last 59.34) - open/close 35733 (time 1788602137.08 total 1210.75 last 66.82) - open/close 36408 (time 1788602147.09 total 1220.76 last 67.43) - open/close 37106 (time 1788602157.10 total 1230.78 last 69.71) - open/close 37754 (time 1788602167.11 total 1240.79 last 64.72) - open/close 38453 (time 1788602177.14 total 1250.81 last 69.74) - open/close 39068 (time 1788602187.14 total 1260.81 last 61.49) - open/close 39745 (time 1788602197.14 total 1270.82 last 67.67) - open/close 40000 (time 1788602201.38 total 1275.06 last 60.11) - open/close 40617 (time 1788602211.39 total 1285.07 last 61.67) - open/close 41298 (time 1788602221.40 total 1295.08 last 68.01) - open/close 41979 (time 1788602231.42 total 1305.10 last 67.97) - open/close 42673 (time 1788602241.43 total 1315.10 last 69.36) - open/close 43336 (time 1788602251.44 total 1325.12 last 66.19) - open/close 43956 (time 1788602261.45 total 1335.13 last 61.93) - open/close 44559 (time 1788602271.45 total 1345.13 last 60.30) - open/close 45223 (time 1788602281.46 total 1355.14 last 66.37) - open/close 45805 (time 1788602291.46 total 1365.14 last 58.17) - open/close 46420 (time 1788602301.47 total 1375.14 last 61.48) - open/close 47071 (time 1788602311.47 total 1385.15 last 65.08) - open/close 47661 (time 1788602321.47 total 1395.15 last 58.99) - open/close 48265 (time 1788602331.48 total 1405.16 last 60.32) - open/close 48913 (time 1788602341.50 total 1415.18 last 64.69) - open/close 49548 (time 1788602351.53 total 1425.21 last 63.32) total: 50000 open/close in 1432.08 seconds: 34.91 ops/second - unlinked 0 (time 1788602363 ; total 0 ; last 0) - unlinked 10000 (time 1788602520 ; total 157 ; last 157) - unlinked 20000 (time 1788602681 ; total 318 ; last 161) - unlinked 30000 (time 1788602845 ; total 482 ; last 164) - unlinked 40000 (time 1788603003 ; total 640 ; last 158) total: 50000 unlinks in 785 seconds: 63.694267 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg334-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg334-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg334-server stop mds service on oleg334-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg334-server unloading modules via unload_modules_local on: 'oleg334-server' oleg334-server: oleg334-server.virtnet: executing unload_modules_local oleg334-server: modules unloaded. pdsh@oleg334-client: oleg334-server: ssh exited with exit code 2 pdsh@oleg334-client: oleg334-client: ssh exited with exit code 2 pdsh@oleg334-client: oleg334-client: ssh exited with exit code 2 pdsh@oleg334-client: oleg334-server: ssh exited with exit code 2