== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:52:53 (1780336373) start mds service on oleg225-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg225-server' oleg225-server: oleg225-server.virtnet: executing load_modules_local oleg225-server: Loading modules from /home/green/git/lustre-release/lustre oleg225-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg225-server: detected 4 online CPUs by sysfs oleg225-server: MODOPTS_LIBCFS= oleg225-server: Force libcfs to create 2 CPU partitions oleg225-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg225-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg225-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg225-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Started lustre-OST0000 oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg225-client.virtnet: -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 161 (time 1780336464.44 total 10.03 last 16.06) - open/close 322 (time 1780336474.46 total 20.05 last 16.06) - open/close 429 (time 1780336484.49 total 30.08 last 10.67) - open/close 447 (time 1780336495.47 total 41.06 last 1.64) - open/close 641 (time 1780336505.47 total 51.07 last 19.39) - open/close 829 (time 1780336515.78 total 61.37 last 18.25) - open/close 1003 (time 1780336525.95 total 71.54 last 17.11) - open/close 1195 (time 1780336536.96 total 82.56 last 17.43) - open/close 1394 (time 1780336546.97 total 92.56 last 19.89) - open/close 1487 (time 1780336558.47 total 104.06 last 8.09) - open/close 1646 (time 1780336568.58 total 114.17 last 15.73) - open/close 1882 (time 1780336579.66 total 125.25 last 21.31) - open/close 2043 (time 1780336589.95 total 135.54 last 15.64) - open/close 2279 (time 1780336600.01 total 145.60 last 23.47) - open/close 2474 (time 1780336610.29 total 155.88 last 18.96) - open/close 2531 (time 1780336621.33 total 166.92 last 5.17) - open/close 2736 (time 1780336631.34 total 176.94 last 20.46) - open/close 2953 (time 1780336642.13 total 187.72 last 20.12) - open/close 3132 (time 1780336653.11 total 198.70 last 16.31) - open/close 3227 (time 1780336664.18 total 209.77 last 8.58) - open/close 3281 (time 1780336675.00 total 220.59 last 4.99) - open/close 3420 (time 1780336685.80 total 231.39 last 12.88) - open/close 3654 (time 1780336696.86 total 242.45 last 21.16) - open/close 3890 (time 1780336707.19 total 252.78 last 22.84) - open/close 4143 (time 1780336717.43 total 263.02 last 24.72) - open/close 4298 (time 1780336728.18 total 273.77 last 14.41) - open/close 4466 (time 1780336738.99 total 284.58 last 15.54) - open/close 4612 (time 1780336749.56 total 295.15 last 13.82) - open/close 4753 (time 1780336760.05 total 305.64 last 13.44) - open/close 4813 (time 1780336770.42 total 316.01 last 5.78) - open/close 4915 (time 1780336781.27 total 326.86 last 9.40) - open/close 5037 (time 1780336791.87 total 337.46 last 11.52) - open/close 5172 (time 1780336801.90 total 347.49 last 13.45) - open/close 5283 (time 1780336812.88 total 358.47 last 10.11) - open/close 5378 (time 1780336822.95 total 368.54 last 9.43) - open/close 5501 (time 1780336834.81 total 380.40 last 10.38) - open/close 5701 (time 1780336845.17 total 390.76 last 19.30) - open/close 5859 (time 1780336855.59 total 401.18 last 15.16) - open/close 6074 (time 1780336865.59 total 411.18 last 21.49) - open/close 6238 (time 1780336877.25 total 422.84 last 14.07) - open/close 6464 (time 1780336887.25 total 432.85 last 22.60) - open/close 6700 (time 1780336897.27 total 442.86 last 23.56) - open/close 6851 (time 1780336908.00 total 453.59 last 14.08) - open/close 6950 (time 1780336918.01 total 463.60 last 9.89) - open/close 7093 (time 1780336928.65 total 474.24 last 13.44) - open/close 7212 (time 1780336939.46 total 485.05 last 11.01) - open/close 7417 (time 1780336950.38 total 495.97 last 18.76) - open/close 7651 (time 1780336961.32 total 506.91 last 21.40) - open/close 7900 (time 1780336971.32 total 516.91 last 24.89) - open/close 8055 (time 1780336981.32 total 526.91 last 15.50) - open/close 8182 (time 1780336991.35 total 536.94 last 12.67) - open/close 8412 (time 1780337001.84 total 547.43 last 21.92) - open/close 8579 (time 1780337012.41 total 558.00 last 15.80) - open/close 8799 (time 1780337023.36 total 568.95 last 20.09) - open/close 9007 (time 1780337034.07 total 579.66 last 19.43) - open/close 9234 (time 1780337044.09 total 589.68 last 22.66) - open/close 9406 (time 1780337054.09 total 599.68 last 17.19) - open/close 9617 (time 1780337065.14 total 610.73 last 19.11) - open/close 9806 (time 1780337075.95 total 621.54 last 17.48) - open/close 9903 (time 1780337087.45 total 633.04 last 8.44) - open/close 10000 (time 1780337093.95 total 639.54 last 14.92) - open/close 10136 (time 1780337105.01 total 650.60 last 12.29) - open/close 10309 (time 1780337115.03 total 660.62 last 17.26) - open/close 10419 (time 1780337125.04 total 670.63 last 10.99) - open/close 10641 (time 1780337135.75 total 681.34 last 20.73) - open/close 10876 (time 1780337145.77 total 691.36 last 23.46) - open/close 11005 (time 1780337155.92 total 701.51 last 12.70) - open/close 11123 (time 1780337165.93 total 711.52 last 11.79) - open/close 11279 (time 1780337175.94 total 721.53 last 15.59) - open/close 11454 (time 1780337185.96 total 731.55 last 17.46) - open/close 11705 (time 1780337196.74 total 742.33 last 23.28) - open/close 11986 (time 1780337206.76 total 752.35 last 28.06) - open/close 12192 (time 1780337219.14 total 764.73 last 16.63) - open/close 12463 (time 1780337230.02 total 775.61 last 24.91) - open/close 12692 (time 1780337240.92 total 786.51 last 21.01) - open/close 12850 (time 1780337250.92 total 796.51 last 15.80) - open/close 13037 (time 1780337262.22 total 807.81 last 16.55) - open/close 13267 (time 1780337272.23 total 817.82 last 22.97) - open/close 13439 (time 1780337282.75 total 828.34 last 16.36) - open/close 13614 (time 1780337292.76 total 838.35 last 17.47) - open/close 13876 (time 1780337302.77 total 848.36 last 26.18) - open/close 14158 (time 1780337312.82 total 858.41 last 28.07) - open/close 14379 (time 1780337322.82 total 868.41 last 22.10) - open/close 14446 (time 1780337333.10 total 878.69 last 6.52) - open/close 14626 (time 1780337344.12 total 889.71 last 16.33) - open/close 14827 (time 1780337354.44 total 900.03 last 19.48) - open/close 15008 (time 1780337364.45 total 910.04 last 18.08) - open/close 15279 (time 1780337375.48 total 921.07 last 24.57) - open/close 15504 (time 1780337385.96 total 931.55 last 21.48) - open/close 15712 (time 1780337395.96 total 941.55 last 20.79) - open/close 15920 (time 1780337405.97 total 951.56 last 20.78) - open/close 16045 (time 1780337416.11 total 961.70 last 12.34) - open/close 16264 (time 1780337426.12 total 971.72 last 21.86) - open/close 16491 (time 1780337436.14 total 981.73 last 22.66) - open/close 16670 (time 1780337446.27 total 991.86 last 17.67) - open/close 16832 (time 1780337456.31 total 1001.90 last 16.14) - open/close 17004 (time 1780337466.35 total 1011.94 last 17.14) - open/close 17182 (time 1780337476.49 total 1022.08 last 17.55) - open/close 17458 (time 1780337486.49 total 1032.08 last 27.59) - open/close 17944 (time 1780337496.50 total 1042.09 last 48.56) - open/close 18478 (time 1780337506.52 total 1052.11 last 53.32) - open/close 19055 (time 1780337516.54 total 1062.13 last 57.58) - open/close 19704 (time 1780337526.55 total 1072.14 last 64.84) - open/close 20000 (time 1780337530.24 total 1075.83 last 80.15) - open/close 20765 (time 1780337540.24 total 1085.83 last 76.47) - open/close 21478 (time 1780337550.25 total 1095.84 last 71.22) - open/close 22154 (time 1780337560.27 total 1105.86 last 67.48) - open/close 22820 (time 1780337570.28 total 1115.87 last 66.57) - open/close 23551 (time 1780337580.30 total 1125.89 last 72.96) - open/close 24229 (time 1780337590.30 total 1135.89 last 67.80) - open/close 25038 (time 1780337600.31 total 1145.90 last 80.82) - open/close 25804 (time 1780337610.32 total 1155.91 last 76.52) - open/close 26642 (time 1780337620.33 total 1165.92 last 83.69) - open/close 27381 (time 1780337630.34 total 1175.93 last 73.81) - open/close 28222 (time 1780337640.36 total 1185.95 last 83.94) - open/close 28905 (time 1780337650.37 total 1195.96 last 68.26) - open/close 29406 (time 1780337660.39 total 1205.98 last 49.98) - open/close 29919 (time 1780337670.39 total 1215.98 last 51.29) - open/close 30546 (time 1780337680.40 total 1225.99 last 62.64) - open/close 31176 (time 1780337690.41 total 1236.01 last 62.93) - open/close 31909 (time 1780337700.42 total 1246.01 last 73.29) - open/close 32672 (time 1780337710.42 total 1256.01 last 76.28) - open/close 33425 (time 1780337720.43 total 1266.02 last 75.20) - open/close 34071 (time 1780337730.43 total 1276.02 last 64.60) - open/close 34753 (time 1780337740.43 total 1286.02 last 68.19) - open/close 35503 (time 1780337750.44 total 1296.03 last 74.94) - open/close 36147 (time 1780337760.45 total 1306.04 last 64.35) - open/close 36783 (time 1780337770.47 total 1316.06 last 63.50) - open/close 37496 (time 1780337780.47 total 1326.06 last 71.28) - open/close 38217 (time 1780337790.47 total 1336.06 last 72.07) - open/close 38908 (time 1780337800.48 total 1346.07 last 69.03) - open/close 39601 (time 1780337810.49 total 1356.09 last 69.20) - open/close 40000 (time 1780337815.63 total 1361.22 last 77.71) - open/close 40631 (time 1780337825.63 total 1371.22 last 63.07) - open/close 41252 (time 1780337835.63 total 1381.22 last 62.10) - open/close 41955 (time 1780337845.63 total 1391.23 last 70.29) - open/close 42553 (time 1780337855.65 total 1401.24 last 59.73) - open/close 43242 (time 1780337865.66 total 1411.25 last 68.79) - open/close 43915 (time 1780337875.68 total 1421.27 last 67.19) - open/close 44625 (time 1780337885.71 total 1431.30 last 70.78) - open/close 45133 (time 1780337895.72 total 1441.31 last 50.75) - open/close 45589 (time 1780337905.75 total 1451.34 last 45.45) - open/close 46173 (time 1780337915.76 total 1461.35 last 58.36) - open/close 46817 (time 1780337925.77 total 1471.36 last 64.36) - open/close 47472 (time 1780337935.77 total 1481.36 last 65.44) - open/close 48126 (time 1780337945.78 total 1491.37 last 65.36) - open/close 48857 (time 1780337955.80 total 1501.39 last 72.95) - open/close 49525 (time 1780337965.80 total 1511.39 last 66.79) total: 50000 open/close in 1519.84 seconds: 32.90 ops/second - unlinked 0 (time 1780337980 ; total 0 ; last 0) - unlinked 10000 (time 1780338147 ; total 167 ; last 166) - unlinked 20000 (time 1780338308 ; total 328 ; last 161) - unlinked 30000 (time 1780338462 ; total 482 ; last 154) - unlinked 40000 (time 1780338616 ; total 636 ; last 154) total: 50000 unlinks in 789 seconds: 63.371357 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg225-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg225-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server stop mds service on oleg225-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server unloading modules via unload_modules_local on: 'oleg225-server' oleg225-server: oleg225-server.virtnet: executing unload_modules_local oleg225-server: modules unloaded. pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2 pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2 pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2 pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2