== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:56:05 (1788155765) start mds service on oleg147-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg147-server' oleg147-server: oleg147-server.virtnet: executing load_modules_local oleg147-server: Loading modules from /home/green/git/lustre-release/lustre oleg147-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg147-server: detected 4 online CPUs by sysfs oleg147-server: MODOPTS_LIBCFS= oleg147-server: Force libcfs to create 2 CPU partitions oleg147-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg147-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg147-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug -1 all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg147-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg147-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg147-server: oleg147-server.virtnet: executing set_default_debug -1 all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-OST0000 oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg147-client.virtnet: -o user_xattr,flock 192.168.201.147@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 239 (time 1788155852.64 total 10.00 last 23.90) - open/close 384 (time 1788155863.11 total 20.47 last 13.85) - open/close 495 (time 1788155874.14 total 31.50 last 10.07) - open/close 622 (time 1788155884.24 total 41.60 last 12.57) - open/close 833 (time 1788155894.27 total 51.63 last 21.03) - open/close 938 (time 1788155904.32 total 61.68 last 10.45) - open/close 1064 (time 1788155914.34 total 71.70 last 12.58) - open/close 1306 (time 1788155924.68 total 82.04 last 23.39) - open/close 1547 (time 1788155935.30 total 92.66 last 22.70) - open/close 1768 (time 1788155946.00 total 103.36 last 20.66) - open/close 1994 (time 1788155957.03 total 114.38 last 20.50) - open/close 2252 (time 1788155967.03 total 124.39 last 25.79) - open/close 2358 (time 1788155977.03 total 134.39 last 10.59) - open/close 2532 (time 1788155987.07 total 144.43 last 17.34) - open/close 2626 (time 1788155998.50 total 155.86 last 8.22) - open/close 2701 (time 1788156008.51 total 165.87 last 7.50) - open/close 2910 (time 1788156018.51 total 175.87 last 20.90) - open/close 3060 (time 1788156028.53 total 185.89 last 14.97) - open/close 3235 (time 1788156038.62 total 195.98 last 17.34) - open/close 3398 (time 1788156048.65 total 206.01 last 16.26) - open/close 3560 (time 1788156058.85 total 216.21 last 15.88) - open/close 3735 (time 1788156068.87 total 226.23 last 17.47) - open/close 3836 (time 1788156079.37 total 236.73 last 9.62) - open/close 4008 (time 1788156089.39 total 246.74 last 17.17) - open/close 4211 (time 1788156100.43 total 257.79 last 18.37) - open/close 4374 (time 1788156110.46 total 267.82 last 16.26) - open/close 4476 (time 1788156120.47 total 277.83 last 10.19) - open/close 4634 (time 1788156131.21 total 288.57 last 14.71) - open/close 4818 (time 1788156142.03 total 299.39 last 17.01) - open/close 5064 (time 1788156152.40 total 309.75 last 23.73) - open/close 5322 (time 1788156162.49 total 319.85 last 25.55) - open/close 5491 (time 1788156173.44 total 330.80 last 15.43) - open/close 5751 (time 1788156183.47 total 340.83 last 25.92) - open/close 5918 (time 1788156199.45 total 356.81 last 10.46) - open/close 6109 (time 1788156210.14 total 367.50 last 17.86) - open/close 6252 (time 1788156220.19 total 377.55 last 14.23) - open/close 6408 (time 1788156230.35 total 387.71 last 15.36) - open/close 6476 (time 1788156240.36 total 397.72 last 6.79) - open/close 6670 (time 1788156251.35 total 408.71 last 17.65) - open/close 6853 (time 1788156261.59 total 418.95 last 17.88) - open/close 7007 (time 1788156272.28 total 429.64 last 14.40) - open/close 7241 (time 1788156282.41 total 439.77 last 23.11) - open/close 7296 (time 1788156293.08 total 450.44 last 5.15) - open/close 7444 (time 1788156303.95 total 461.31 last 13.61) - open/close 7689 (time 1788156314.38 total 471.74 last 23.50) - open/close 7906 (time 1788156324.39 total 481.75 last 21.67) - open/close 8158 (time 1788156334.40 total 491.76 last 25.18) - open/close 8265 (time 1788156344.92 total 502.28 last 10.17) - open/close 8459 (time 1788156355.79 total 513.15 last 17.85) - open/close 8616 (time 1788156366.78 total 524.14 last 14.28) - open/close 8879 (time 1788156376.79 total 534.15 last 26.28) - open/close 9054 (time 1788156389.35 total 546.71 last 13.93) - open/close 9293 (time 1788156400.33 total 557.69 last 21.77) - open/close 9405 (time 1788156410.38 total 567.74 last 11.15) - open/close 9505 (time 1788156420.39 total 577.75 last 9.99) - open/close 9624 (time 1788156430.67 total 588.03 last 11.57) - open/close 9757 (time 1788156440.78 total 598.13 last 13.17) - open/close 9907 (time 1788156451.28 total 608.64 last 14.28) - open/close 10000 (time 1788156455.47 total 612.83 last 22.18) - open/close 10111 (time 1788156465.48 total 622.84 last 11.09) - open/close 10318 (time 1788156475.49 total 632.85 last 20.68) - open/close 10539 (time 1788156487.82 total 645.18 last 17.93) - open/close 10773 (time 1788156499.25 total 656.61 last 20.47) - open/close 10975 (time 1788156509.51 total 666.87 last 19.68) - open/close 11238 (time 1788156519.87 total 677.23 last 25.39) - open/close 11440 (time 1788156529.88 total 687.24 last 20.19) - open/close 11604 (time 1788156540.65 total 698.00 last 15.23) - open/close 11764 (time 1788156550.67 total 708.03 last 15.95) - open/close 11936 (time 1788156560.82 total 718.17 last 16.96) - open/close 12101 (time 1788156571.03 total 728.39 last 16.16) - open/close 12255 (time 1788156581.97 total 739.32 last 14.08) - open/close 12342 (time 1788156591.97 total 749.33 last 8.69) - open/close 12537 (time 1788156602.82 total 760.18 last 17.97) - open/close 12686 (time 1788156613.21 total 770.57 last 14.34) - open/close 12820 (time 1788156623.82 total 781.18 last 12.64) - open/close 12996 (time 1788156633.88 total 791.24 last 17.50) - open/close 13209 (time 1788156644.07 total 801.43 last 20.90) - open/close 13387 (time 1788156654.22 total 811.58 last 17.53) - open/close 13528 (time 1788156666.46 total 823.82 last 11.53) - open/close 13666 (time 1788156677.23 total 834.59 last 12.81) - open/close 13805 (time 1788156687.25 total 844.61 last 13.87) - open/close 14008 (time 1788156697.34 total 854.70 last 20.11) - open/close 14235 (time 1788156708.25 total 865.61 last 20.82) - open/close 14399 (time 1788156718.61 total 875.96 last 15.83) - open/close 14663 (time 1788156729.49 total 886.85 last 24.25) - open/close 14899 (time 1788156740.24 total 897.60 last 21.95) - open/close 14997 (time 1788156750.86 total 908.22 last 9.23) - open/close 15164 (time 1788156762.98 total 920.34 last 13.78) - open/close 15373 (time 1788156775.91 total 933.27 last 16.17) - open/close 15589 (time 1788156785.95 total 943.31 last 21.51) - open/close 15732 (time 1788156796.86 total 954.22 last 13.10) - open/close 15956 (time 1788156809.09 total 966.45 last 18.33) - open/close 16212 (time 1788156819.09 total 976.45 last 25.59) - open/close 16399 (time 1788156829.98 total 987.34 last 17.17) - open/close 16614 (time 1788156839.99 total 997.35 last 21.49) - open/close 16766 (time 1788156850.01 total 1007.37 last 15.16) - open/close 16851 (time 1788156860.37 total 1017.73 last 8.21) - open/close 17033 (time 1788156870.41 total 1027.77 last 18.12) - open/close 17153 (time 1788156880.49 total 1037.85 last 11.91) - open/close 17248 (time 1788156890.89 total 1048.25 last 9.13) - open/close 17343 (time 1788156901.84 total 1059.20 last 8.67) - open/close 17842 (time 1788156911.85 total 1069.21 last 49.89) - open/close 18410 (time 1788156921.85 total 1079.21 last 56.79) - open/close 19124 (time 1788156931.86 total 1089.22 last 71.34) - open/close 19870 (time 1788156941.86 total 1099.22 last 74.54) - open/close 20000 (time 1788156944.07 total 1101.43 last 58.94) - open/close 20699 (time 1788156954.08 total 1111.44 last 69.85) - open/close 21377 (time 1788156964.10 total 1121.46 last 67.68) - open/close 21964 (time 1788156974.10 total 1131.46 last 58.69) - open/close 22507 (time 1788156984.10 total 1141.46 last 54.27) - open/close 23164 (time 1788156994.11 total 1151.47 last 65.64) - open/close 23857 (time 1788157004.12 total 1161.48 last 69.25) - open/close 24444 (time 1788157014.13 total 1171.49 last 58.63) - open/close 24922 (time 1788157024.14 total 1181.50 last 47.75) - open/close 25456 (time 1788157034.17 total 1191.53 last 53.24) - open/close 26010 (time 1788157044.19 total 1201.55 last 55.28) - open/close 26520 (time 1788157054.20 total 1211.55 last 50.99) - open/close 27090 (time 1788157064.22 total 1221.58 last 56.86) - open/close 27778 (time 1788157074.22 total 1231.58 last 68.80) - open/close 28432 (time 1788157084.23 total 1241.59 last 65.33) - open/close 28969 (time 1788157094.23 total 1251.59 last 53.69) - open/close 29526 (time 1788157104.26 total 1261.61 last 55.58) - open/close 30000 (time 1788157111.34 total 1268.70 last 66.87) - open/close 30503 (time 1788157121.35 total 1278.71 last 50.25) - open/close 31066 (time 1788157131.36 total 1288.72 last 56.27) - open/close 31600 (time 1788157141.38 total 1298.74 last 53.30) - open/close 32179 (time 1788157151.41 total 1308.77 last 57.73) - open/close 32854 (time 1788157161.45 total 1318.81 last 67.22) - open/close 33381 (time 1788157171.45 total 1328.81 last 52.70) - open/close 34012 (time 1788157181.45 total 1338.81 last 63.09) - open/close 34663 (time 1788157191.47 total 1348.83 last 64.96) - open/close 35308 (time 1788157201.48 total 1358.84 last 64.44) - open/close 35935 (time 1788157211.48 total 1368.84 last 62.68) - open/close 36531 (time 1788157221.50 total 1378.86 last 59.53) - open/close 37256 (time 1788157231.51 total 1388.87 last 72.42) - open/close 37976 (time 1788157241.51 total 1398.87 last 71.97) - open/close 38668 (time 1788157251.54 total 1408.90 last 69.02) - open/close 39299 (time 1788157261.55 total 1418.91 last 63.04) - open/close 39853 (time 1788157271.57 total 1428.93 last 55.27) - open/close 40000 (time 1788157273.78 total 1431.14 last 66.51) - open/close 40474 (time 1788157283.79 total 1441.15 last 47.37) - open/close 40979 (time 1788157293.81 total 1451.17 last 50.38) - open/close 41656 (time 1788157303.82 total 1461.18 last 67.66) - open/close 42364 (time 1788157313.83 total 1471.19 last 70.72) - open/close 43113 (time 1788157323.83 total 1481.19 last 74.87) - open/close 43761 (time 1788157333.84 total 1491.20 last 64.75) - open/close 44443 (time 1788157343.85 total 1501.21 last 68.13) - open/close 45133 (time 1788157353.85 total 1511.21 last 68.97) - open/close 45767 (time 1788157363.86 total 1521.21 last 63.40) - open/close 46485 (time 1788157373.87 total 1531.23 last 71.71) - open/close 47109 (time 1788157383.88 total 1541.24 last 62.35) - open/close 47731 (time 1788157393.88 total 1551.24 last 62.16) - open/close 48358 (time 1788157403.89 total 1561.25 last 62.67) - open/close 48999 (time 1788157413.90 total 1571.26 last 64.02) - open/close 49625 (time 1788157423.91 total 1581.27 last 62.54) total: 50000 open/close in 1587.70 seconds: 31.49 ops/second - unlinked 0 (time 1788157435 ; total 0 ; last 0) - unlinked 10000 (time 1788157607 ; total 172 ; last 172) - unlinked 20000 (time 1788157771 ; total 336 ; last 164) - unlinked 30000 (time 1788157934 ; total 499 ; last 163) - unlinked 40000 (time 1788158094 ; total 659 ; last 160) total: 50000 unlinks in 823 seconds: 60.753342 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg147-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg147-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg147-server stop mds service on oleg147-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg147-server unloading modules via unload_modules_local on: 'oleg147-server' oleg147-server: oleg147-server.virtnet: executing unload_modules_local oleg147-server: modules unloaded. pdsh@oleg147-client: oleg147-server: ssh exited with exit code 2 pdsh@oleg147-client: oleg147-client: ssh exited with exit code 2 pdsh@oleg147-client: oleg147-client: ssh exited with exit code 2 pdsh@oleg147-client: oleg147-server: ssh exited with exit code 2