== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:58:48 (1787367528) start mds service on oleg345-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions oleg345-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg345-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg345-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg345-server: oleg345-server.virtnet: executing set_default_debug -1 all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg345-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg345-server: oleg345-server.virtnet: executing set_default_debug -1 all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-OST0000 oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 207 (time 1787367614.18 total 10.02 last 20.66) - open/close 360 (time 1787367624.37 total 20.21 last 15.02) - open/close 479 (time 1787367634.39 total 30.22 last 11.88) - open/close 570 (time 1787367645.16 total 40.99 last 8.45) - open/close 722 (time 1787367657.33 total 53.17 last 12.49) - open/close 871 (time 1787367668.20 total 64.04 last 13.70) - open/close 953 (time 1787367678.44 total 74.28 last 8.01) - open/close 1183 (time 1787367688.46 total 84.30 last 22.95) - open/close 1390 (time 1787367699.33 total 95.16 last 19.05) - open/close 1519 (time 1787367709.55 total 105.39 last 12.62) - open/close 1712 (time 1787367719.65 total 115.49 last 19.10) - open/close 1883 (time 1787367730.41 total 126.25 last 15.89) - open/close 2049 (time 1787367741.23 total 137.06 last 15.35) - open/close 2315 (time 1787367751.23 total 147.07 last 26.59) - open/close 2517 (time 1787367761.71 total 157.54 last 19.28) - open/close 2658 (time 1787367771.71 total 167.55 last 14.09) - open/close 2857 (time 1787367781.73 total 177.57 last 19.86) - open/close 2966 (time 1787367791.74 total 187.57 last 10.89) - open/close 3097 (time 1787367802.29 total 198.13 last 12.41) - open/close 3260 (time 1787367812.35 total 208.19 last 16.21) - open/close 3439 (time 1787367822.51 total 218.35 last 17.62) - open/close 3604 (time 1787367832.53 total 228.37 last 16.46) - open/close 3766 (time 1787367842.91 total 238.75 last 15.60) - open/close 3966 (time 1787367853.12 total 248.95 last 19.60) - open/close 4165 (time 1787367863.53 total 259.36 last 19.12) - open/close 4265 (time 1787367873.59 total 269.42 last 9.94) - open/close 4402 (time 1787367884.04 total 279.88 last 13.10) - open/close 4574 (time 1787367895.02 total 290.86 last 15.66) - open/close 4666 (time 1787367905.33 total 301.17 last 8.93) - open/close 4821 (time 1787367916.21 total 312.05 last 14.24) - open/close 4924 (time 1787367926.21 total 322.05 last 10.30) - open/close 5078 (time 1787367937.03 total 332.87 last 14.23) - open/close 5216 (time 1787367947.04 total 342.87 last 13.80) - open/close 5389 (time 1787367957.24 total 353.08 last 16.95) - open/close 5568 (time 1787367967.25 total 363.09 last 17.89) - open/close 5661 (time 1787367979.55 total 375.39 last 7.56) - open/close 5845 (time 1787367989.58 total 385.42 last 18.35) - open/close 6054 (time 1787368000.43 total 396.27 last 19.26) - open/close 6293 (time 1787368010.43 total 406.27 last 23.89) - open/close 6530 (time 1787368021.29 total 417.13 last 21.82) - open/close 6813 (time 1787368031.30 total 427.14 last 28.29) - open/close 6951 (time 1787368041.30 total 437.14 last 13.79) - open/close 7162 (time 1787368051.39 total 447.23 last 20.91) - open/close 7340 (time 1787368062.39 total 458.23 last 16.19) - open/close 7544 (time 1787368072.74 total 468.58 last 19.70) - open/close 7805 (time 1787368083.40 total 479.23 last 24.50) - open/close 8047 (time 1787368093.40 total 489.24 last 24.19) - open/close 8254 (time 1787368103.42 total 499.26 last 20.65) - open/close 8484 (time 1787368113.70 total 509.54 last 22.38) - open/close 8738 (time 1787368123.71 total 519.55 last 25.37) - open/close 8889 (time 1787368133.74 total 529.57 last 15.06) - open/close 9130 (time 1787368144.37 total 540.20 last 22.67) - open/close 9385 (time 1787368155.33 total 551.16 last 23.27) - open/close 9619 (time 1787368165.67 total 561.51 last 22.62) - open/close 9887 (time 1787368175.68 total 571.52 last 26.78) - open/close 10000 (time 1787368181.05 total 576.89 last 21.04) - open/close 10212 (time 1787368191.06 total 586.90 last 21.18) - open/close 10397 (time 1787368201.84 total 597.68 last 17.15) - open/close 10590 (time 1787368212.16 total 607.99 last 18.72) - open/close 10721 (time 1787368224.29 total 620.13 last 10.79) - open/close 10910 (time 1787368234.43 total 630.26 last 18.65) - open/close 11150 (time 1787368244.96 total 640.80 last 22.77) - open/close 11388 (time 1787368254.97 total 650.81 last 23.78) - open/close 11601 (time 1787368265.35 total 661.18 last 20.53) - open/close 11839 (time 1787368276.29 total 672.12 last 21.75) - open/close 12105 (time 1787368286.31 total 682.15 last 26.53) - open/close 12279 (time 1787368296.58 total 692.41 last 16.95) - open/close 12527 (time 1787368306.59 total 702.43 last 24.76) - open/close 12658 (time 1787368316.83 total 712.67 last 12.79) - open/close 12876 (time 1787368327.06 total 722.90 last 21.32) - open/close 13065 (time 1787368337.06 total 732.90 last 18.89) - open/close 13328 (time 1787368348.07 total 743.91 last 23.90) - open/close 13574 (time 1787368358.70 total 754.54 last 23.14) - open/close 13725 (time 1787368368.71 total 764.54 last 15.09) - open/close 13947 (time 1787368379.20 total 775.04 last 21.15) - open/close 14155 (time 1787368389.36 total 785.20 last 20.47) - open/close 14394 (time 1787368399.97 total 795.81 last 22.54) - open/close 14616 (time 1787368410.62 total 806.45 last 20.85) - open/close 14840 (time 1787368421.22 total 817.06 last 21.13) - open/close 15056 (time 1787368431.22 total 827.06 last 21.60) - open/close 15282 (time 1787368441.97 total 837.81 last 21.02) - open/close 15462 (time 1787368451.98 total 847.82 last 17.98) - open/close 15704 (time 1787368462.00 total 857.84 last 24.15) - open/close 15904 (time 1787368472.88 total 868.72 last 18.38) - open/close 16148 (time 1787368482.89 total 878.73 last 24.37) - open/close 16317 (time 1787368492.91 total 888.75 last 16.87) - open/close 16553 (time 1787368502.93 total 898.76 last 23.58) - open/close 16753 (time 1787368512.93 total 908.76 last 20.00) - open/close 16954 (time 1787368523.78 total 919.62 last 18.52) - open/close 17218 (time 1787368534.64 total 930.48 last 24.32) - open/close 17458 (time 1787368544.67 total 940.51 last 23.91) - open/close 17688 (time 1787368554.93 total 950.77 last 22.42) - open/close 18161 (time 1787368564.95 total 960.79 last 47.22) - open/close 18844 (time 1787368574.95 total 970.79 last 68.28) - open/close 19521 (time 1787368584.97 total 980.80 last 67.63) - open/close 20000 (time 1787368591.97 total 987.81 last 68.39) - open/close 20685 (time 1787368601.98 total 997.82 last 68.41) - open/close 21425 (time 1787368612.00 total 1007.83 last 73.91) - open/close 22136 (time 1787368622.01 total 1017.84 last 71.02) - open/close 22785 (time 1787368632.02 total 1027.85 last 64.84) - open/close 23407 (time 1787368642.02 total 1037.85 last 62.19) - open/close 24057 (time 1787368652.02 total 1047.86 last 64.96) - open/close 24653 (time 1787368662.05 total 1057.88 last 59.45) - open/close 25247 (time 1787368672.05 total 1067.89 last 59.40) - open/close 25864 (time 1787368682.05 total 1077.89 last 61.66) - open/close 26385 (time 1787368692.07 total 1087.91 last 52.00) - open/close 27000 (time 1787368702.10 total 1097.93 last 61.36) - open/close 27712 (time 1787368712.11 total 1107.94 last 71.14) - open/close 28341 (time 1787368722.11 total 1117.95 last 62.88) - open/close 28995 (time 1787368732.12 total 1127.96 last 65.34) - open/close 29697 (time 1787368742.13 total 1137.96 last 70.15) - open/close 30000 (time 1787368746.69 total 1142.53 last 66.38) - open/close 30661 (time 1787368756.70 total 1152.54 last 66.02) - open/close 31247 (time 1787368766.71 total 1162.54 last 58.58) - open/close 31857 (time 1787368776.71 total 1172.55 last 60.98) - open/close 32535 (time 1787368786.72 total 1182.56 last 67.70) - open/close 33192 (time 1787368796.74 total 1192.57 last 65.61) - open/close 33827 (time 1787368806.74 total 1202.58 last 63.45) - open/close 34483 (time 1787368816.75 total 1212.59 last 65.58) - open/close 35173 (time 1787368826.77 total 1222.61 last 68.82) - open/close 35833 (time 1787368836.78 total 1232.61 last 65.99) - open/close 36487 (time 1787368846.79 total 1242.63 last 65.30) - open/close 37153 (time 1787368856.81 total 1252.65 last 66.47) - open/close 37619 (time 1787368866.82 total 1262.66 last 46.56) - open/close 38064 (time 1787368876.86 total 1272.70 last 44.29) - open/close 38630 (time 1787368886.87 total 1282.71 last 56.56) - open/close 39250 (time 1787368896.88 total 1292.72 last 61.92) - open/close 39887 (time 1787368906.89 total 1302.73 last 63.67) - open/close 40562 (time 1787368916.90 total 1312.74 last 67.44) - open/close 41191 (time 1787368926.90 total 1322.74 last 62.90) - open/close 41809 (time 1787368936.91 total 1332.75 last 61.70) - open/close 42482 (time 1787368946.92 total 1342.76 last 67.26) - open/close 43151 (time 1787368956.93 total 1352.76 last 66.85) - open/close 43748 (time 1787368966.93 total 1362.76 last 59.70) - open/close 44374 (time 1787368976.94 total 1372.77 last 62.54) - open/close 44872 (time 1787368986.94 total 1382.78 last 49.76) - open/close 45431 (time 1787368996.95 total 1392.78 last 55.89) - open/close 46048 (time 1787369006.95 total 1402.79 last 61.68) - open/close 46661 (time 1787369016.96 total 1412.80 last 61.25) - open/close 47281 (time 1787369026.96 total 1422.80 last 61.98) - open/close 47937 (time 1787369037.00 total 1432.84 last 65.36) - open/close 48631 (time 1787369047.00 total 1442.84 last 69.38) - open/close 49161 (time 1787369057.00 total 1452.84 last 52.99) - open/close 49778 (time 1787369067.01 total 1462.84 last 61.68) total: 50000 open/close in 1466.20 seconds: 34.10 ops/second - unlinked 0 (time 1787369075 ; total 0 ; last 0) - unlinked 10000 (time 1787369236 ; total 161 ; last 161) - unlinked 20000 (time 1787369396 ; total 321 ; last 160) - unlinked 30000 (time 1787369556 ; total 481 ; last 160) - unlinked 40000 (time 1787369712 ; total 637 ; last 156) total: 50000 unlinks in 795 seconds: 62.893082 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg345-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg345-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg345-server stop mds service on oleg345-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg345-server unloading modules via unload_modules_local on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing unload_modules_local oleg345-server: modules unloaded. pdsh@oleg345-client: oleg345-client: ssh exited with exit code 2 pdsh@oleg345-client: oleg345-server: ssh exited with exit code 2 pdsh@oleg345-client: oleg345-client: ssh exited with exit code 2 pdsh@oleg345-client: oleg345-server: ssh exited with exit code 2