== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 11:30:52 (1787585452) start mds service on oleg634-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg634-server' oleg634-server: oleg634-server.virtnet: executing load_modules_local oleg634-server: Loading modules from /home/green/git/lustre-release/lustre oleg634-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg634-server: detected 4 online CPUs by sysfs oleg634-server: MODOPTS_LIBCFS= oleg634-server: Force libcfs to create 2 CPU partitions oleg634-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg634-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg634-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg634-server: oleg634-server.virtnet: executing set_default_debug -1 all pdsh@oleg634-client: oleg634-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg634-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg634-server: oleg634-server.virtnet: executing set_default_debug -1 all pdsh@oleg634-client: oleg634-server: ssh exited with exit code 1 Started lustre-OST0000 oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg634-client.virtnet: -o user_xattr,flock 192.168.206.134@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 95 (time 1787585531.82 total 10.84 last 8.76) - open/close 191 (time 1787585543.31 total 22.34 last 8.35) - open/close 317 (time 1787585553.34 total 32.36 last 12.57) - open/close 471 (time 1787585566.28 total 45.31 last 11.89) - open/close 575 (time 1787585576.74 total 55.77 last 9.94) - open/close 752 (time 1787585586.77 total 65.80 last 17.65) - open/close 956 (time 1787585596.78 total 75.80 last 20.39) - open/close 1103 (time 1787585606.80 total 85.82 last 14.67) - open/close 1269 (time 1787585619.66 total 98.68 last 12.90) - open/close 1399 (time 1787585630.44 total 109.46 last 12.06) - open/close 1562 (time 1787585640.45 total 119.48 last 16.27) - open/close 1754 (time 1787585650.46 total 129.48 last 19.19) - open/close 1978 (time 1787585660.64 total 139.67 last 22.00) - open/close 2197 (time 1787585672.13 total 151.15 last 19.07) - open/close 2430 (time 1787585682.63 total 161.65 last 22.19) - open/close 2702 (time 1787585692.63 total 171.65 last 27.20) - open/close 2880 (time 1787585704.26 total 183.28 last 15.30) - open/close 3116 (time 1787585715.27 total 194.30 last 21.43) - open/close 3403 (time 1787585725.28 total 204.30 last 28.68) - open/close 3684 (time 1787585735.29 total 214.32 last 28.06) - open/close 3889 (time 1787585745.31 total 224.33 last 20.47) - open/close 4023 (time 1787585756.36 total 235.39 last 12.12) - open/close 4161 (time 1787585766.37 total 245.39 last 13.79) - open/close 4354 (time 1787585776.52 total 255.54 last 19.01) - open/close 4584 (time 1787585787.03 total 266.06 last 21.87) - open/close 4815 (time 1787585797.04 total 276.07 last 23.07) - open/close 5054 (time 1787585807.82 total 286.84 last 22.18) - open/close 5293 (time 1787585818.46 total 297.49 last 22.45) - open/close 5533 (time 1787585828.48 total 307.51 last 23.96) - open/close 5763 (time 1787585838.49 total 317.51 last 22.99) - open/close 5989 (time 1787585849.11 total 328.13 last 21.28) - open/close 6227 (time 1787585859.12 total 338.14 last 23.77) - open/close 6359 (time 1787585869.17 total 348.20 last 13.13) - open/close 6435 (time 1787585879.19 total 358.21 last 7.59) - open/close 6578 (time 1787585889.61 total 368.64 last 13.71) - open/close 6716 (time 1787585900.61 total 379.63 last 12.55) - open/close 6821 (time 1787585911.16 total 390.18 last 9.95) - open/close 7051 (time 1787585921.29 total 400.32 last 22.70) - open/close 7223 (time 1787585931.32 total 410.34 last 17.16) - open/close 7307 (time 1787585941.34 total 420.36 last 8.38) - open/close 7560 (time 1787585951.34 total 430.36 last 25.30) - open/close 7722 (time 1787585962.08 total 441.11 last 15.08) - open/close 7902 (time 1787585972.50 total 451.52 last 17.28) - open/close 8083 (time 1787585983.12 total 462.15 last 17.04) - open/close 8215 (time 1787585995.47 total 474.49 last 10.69) - open/close 8336 (time 1787586005.50 total 484.53 last 12.06) - open/close 8396 (time 1787586015.52 total 494.55 last 5.99) - open/close 8501 (time 1787586026.32 total 505.34 last 9.73) - open/close 8686 (time 1787586036.32 total 515.35 last 18.49) - open/close 8890 (time 1787586046.54 total 525.57 last 19.96) - open/close 9131 (time 1787586057.41 total 536.43 last 22.18) - open/close 9376 (time 1787586067.43 total 546.45 last 24.45) - open/close 9604 (time 1787586077.44 total 556.47 last 22.76) - open/close 9804 (time 1787586087.45 total 566.48 last 19.98) - open/close 10000 (time 1787586097.49 total 576.51 last 19.53) - open/close 10237 (time 1787586108.20 total 587.22 last 22.13) - open/close 10440 (time 1787586118.21 total 597.24 last 20.27) - open/close 10691 (time 1787586128.22 total 607.24 last 25.08) - open/close 10863 (time 1787586138.25 total 617.27 last 17.15) - open/close 10988 (time 1787586148.33 total 627.35 last 12.40) - open/close 11110 (time 1787586159.31 total 638.34 last 11.10) - open/close 11308 (time 1787586169.84 total 648.86 last 18.81) - open/close 11443 (time 1787586179.84 total 658.87 last 13.50) - open/close 11627 (time 1787586190.35 total 669.38 last 17.51) - open/close 11858 (time 1787586200.99 total 680.01 last 21.72) - open/close 12087 (time 1787586211.48 total 690.51 last 21.82) - open/close 12340 (time 1787586222.34 total 701.36 last 23.30) - open/close 12590 (time 1787586232.35 total 711.37 last 24.98) - open/close 12736 (time 1787586242.58 total 721.61 last 14.27) - open/close 12816 (time 1787586252.95 total 731.98 last 7.71) - open/close 12889 (time 1787586263.00 total 742.02 last 7.27) - open/close 13071 (time 1787586273.94 total 752.97 last 16.63) - open/close 13318 (time 1787586284.74 total 763.77 last 22.87) - open/close 13604 (time 1787586294.77 total 773.79 last 28.54) - open/close 13776 (time 1787586307.55 total 786.57 last 13.46) - open/close 13983 (time 1787586318.24 total 797.26 last 19.36) - open/close 14166 (time 1787586328.25 total 807.27 last 18.28) - open/close 14275 (time 1787586340.91 total 819.94 last 8.60) - open/close 14483 (time 1787586352.34 total 831.36 last 18.21) - open/close 14715 (time 1787586362.89 total 841.91 last 21.99) - open/close 14916 (time 1787586372.89 total 851.92 last 20.09) - open/close 15072 (time 1787586383.13 total 862.15 last 15.24) - open/close 15350 (time 1787586393.14 total 872.16 last 27.78) - open/close 15533 (time 1787586403.15 total 882.17 last 18.28) - open/close 15761 (time 1787586413.71 total 892.74 last 21.58) - open/close 15967 (time 1787586424.60 total 903.62 last 18.93) - open/close 16176 (time 1787586434.62 total 913.65 last 20.84) - open/close 16404 (time 1787586445.45 total 924.47 last 21.06) - open/close 16599 (time 1787586455.46 total 934.48 last 19.49) - open/close 16853 (time 1787586465.93 total 944.95 last 24.26) - open/close 17077 (time 1787586475.93 total 954.95 last 22.39) - open/close 17325 (time 1787586486.63 total 965.65 last 23.18) - open/close 17584 (time 1787586497.17 total 976.19 last 24.58) - open/close 17805 (time 1787586507.40 total 986.43 last 21.59) - open/close 18322 (time 1787586517.40 total 996.43 last 51.69) - open/close 18927 (time 1787586527.41 total 1006.43 last 60.45) - open/close 19593 (time 1787586537.44 total 1016.46 last 66.41) - open/close 20000 (time 1787586543.72 total 1022.75 last 64.79) - open/close 20605 (time 1787586553.74 total 1032.76 last 60.40) - open/close 21214 (time 1787586563.75 total 1042.77 last 60.82) - open/close 21783 (time 1787586573.75 total 1052.78 last 56.89) - open/close 22465 (time 1787586583.75 total 1062.78 last 68.18) - open/close 23200 (time 1787586593.76 total 1072.79 last 73.45) - open/close 23874 (time 1787586603.76 total 1082.79 last 67.38) - open/close 24568 (time 1787586613.77 total 1092.80 last 69.34) - open/close 25188 (time 1787586623.78 total 1102.80 last 61.96) - open/close 25869 (time 1787586633.78 total 1112.81 last 68.09) - open/close 26574 (time 1787586643.79 total 1122.81 last 70.47) - open/close 27298 (time 1787586653.80 total 1132.82 last 72.33) - open/close 27977 (time 1787586663.80 total 1142.83 last 67.85) - open/close 28616 (time 1787586673.81 total 1152.84 last 63.85) - open/close 29306 (time 1787586683.81 total 1162.84 last 69.00) - open/close 29973 (time 1787586693.82 total 1172.84 last 66.64) - open/close 30692 (time 1787586703.82 total 1182.85 last 71.87) - open/close 31351 (time 1787586713.82 total 1192.85 last 65.89) - open/close 31935 (time 1787586723.83 total 1202.85 last 58.39) - open/close 32527 (time 1787586733.83 total 1212.86 last 59.15) - open/close 33158 (time 1787586743.84 total 1222.87 last 63.05) - open/close 33804 (time 1787586753.85 total 1232.88 last 64.52) - open/close 34476 (time 1787586763.85 total 1242.88 last 67.20) - open/close 35133 (time 1787586773.86 total 1252.89 last 65.63) - open/close 35827 (time 1787586783.87 total 1262.90 last 69.34) - open/close 36535 (time 1787586793.88 total 1272.90 last 70.76) - open/close 37237 (time 1787586803.89 total 1282.91 last 70.13) - open/close 37893 (time 1787586813.91 total 1292.93 last 65.46) - open/close 38578 (time 1787586823.93 total 1302.95 last 68.38) - open/close 39225 (time 1787586833.93 total 1312.96 last 64.68) - open/close 39906 (time 1787586843.94 total 1322.97 last 68.03) - open/close 40560 (time 1787586853.96 total 1332.99 last 65.26) - open/close 41236 (time 1787586863.97 total 1342.99 last 67.58) - open/close 41900 (time 1787586873.97 total 1353.00 last 66.37) - open/close 42635 (time 1787586883.98 total 1363.00 last 73.43) - open/close 43280 (time 1787586893.99 total 1373.01 last 64.45) - open/close 44001 (time 1787586904.00 total 1383.03 last 71.97) - open/close 44617 (time 1787586914.01 total 1393.03 last 61.59) - open/close 45314 (time 1787586924.02 total 1403.05 last 69.58) - open/close 46027 (time 1787586934.03 total 1413.05 last 71.28) - open/close 46650 (time 1787586944.04 total 1423.06 last 62.24) - open/close 47292 (time 1787586954.05 total 1433.07 last 64.11) - open/close 47897 (time 1787586964.07 total 1443.09 last 60.39) - open/close 48470 (time 1787586974.10 total 1453.12 last 57.13) - open/close 49006 (time 1787586984.11 total 1463.13 last 53.56) - open/close 49587 (time 1787586994.12 total 1473.15 last 58.00) total: 50000 open/close in 1479.95 seconds: 33.79 ops/second - unlinked 0 (time 1787587005 ; total 0 ; last 0) - unlinked 10000 (time 1787587169 ; total 164 ; last 164) - unlinked 20000 (time 1787587334 ; total 329 ; last 165) - unlinked 30000 (time 1787587486 ; total 481 ; last 152) - unlinked 40000 (time 1787587655 ; total 650 ; last 169) total: 50000 unlinks in 805 seconds: 62.111801 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg634-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg634-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg634-server stop mds service on oleg634-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg634-server unloading modules via unload_modules_local on: 'oleg634-server' oleg634-server: oleg634-server.virtnet: executing unload_modules_local oleg634-server: modules unloaded. pdsh@oleg634-client: oleg634-client: ssh exited with exit code 2 pdsh@oleg634-client: oleg634-server: ssh exited with exit code 2 pdsh@oleg634-client: oleg634-client: ssh exited with exit code 2 pdsh@oleg634-client: oleg634-server: ssh exited with exit code 2