== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:10:06 (1788156606) start mds service on oleg345-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions oleg345-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg345-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg345-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg345-server: oleg345-server.virtnet: executing set_default_debug -1 all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg345-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg345-server: oleg345-server.virtnet: executing set_default_debug -1 all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-OST0000 oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 243 (time 1788156686.56 total 11.08 last 21.93) - open/close 435 (time 1788156696.56 total 21.09 last 19.19) - open/close 663 (time 1788156706.57 total 31.10 last 22.78) - open/close 906 (time 1788156716.57 total 41.10 last 24.29) - open/close 1123 (time 1788156726.74 total 51.26 last 21.35) - open/close 1351 (time 1788156737.06 total 61.58 last 22.09) - open/close 1590 (time 1788156747.06 total 71.59 last 23.89) - open/close 1843 (time 1788156757.94 total 82.47 last 23.25) - open/close 2052 (time 1788156768.02 total 92.55 last 20.74) - open/close 2239 (time 1788156778.03 total 102.55 last 18.69) - open/close 2364 (time 1788156788.03 total 112.56 last 12.49) - open/close 2490 (time 1788156798.10 total 122.63 last 12.51) - open/close 2705 (time 1788156808.11 total 132.64 last 21.49) - open/close 2896 (time 1788156818.12 total 142.65 last 19.08) - open/close 3173 (time 1788156829.01 total 153.54 last 25.43) - open/close 3261 (time 1788156839.02 total 163.55 last 8.79) - open/close 3389 (time 1788156849.03 total 173.55 last 12.79) - open/close 3510 (time 1788156859.11 total 183.64 last 12.00) - open/close 3702 (time 1788156869.12 total 193.65 last 19.18) - open/close 3972 (time 1788156879.14 total 203.67 last 26.95) - open/close 4203 (time 1788156889.82 total 214.35 last 21.63) - open/close 4398 (time 1788156900.69 total 225.22 last 17.95) - open/close 4634 (time 1788156911.06 total 235.59 last 22.76) - open/close 4858 (time 1788156921.21 total 245.74 last 22.06) - open/close 4962 (time 1788156931.62 total 256.15 last 9.99) - open/close 5202 (time 1788156942.32 total 266.84 last 22.44) - open/close 5461 (time 1788156953.11 total 277.64 last 23.99) - open/close 5642 (time 1788156964.75 total 289.27 last 15.56) - open/close 5777 (time 1788156975.05 total 299.58 last 13.10) - open/close 5932 (time 1788156985.07 total 309.60 last 15.47) - open/close 6051 (time 1788156996.04 total 320.57 last 10.85) - open/close 6171 (time 1788157006.30 total 330.82 last 11.70) - open/close 6332 (time 1788157016.31 total 340.83 last 16.08) - open/close 6459 (time 1788157027.22 total 351.74 last 11.64) - open/close 6589 (time 1788157037.66 total 362.19 last 12.45) - open/close 6762 (time 1788157047.67 total 372.20 last 17.29) - open/close 6983 (time 1788157058.73 total 383.25 last 19.99) - open/close 7171 (time 1788157068.74 total 393.26 last 18.78) - open/close 7294 (time 1788157078.75 total 403.27 last 12.29) - open/close 7438 (time 1788157088.76 total 413.29 last 14.38) - open/close 7549 (time 1788157099.60 total 424.13 last 10.23) - open/close 7676 (time 1788157109.79 total 434.32 last 12.47) - open/close 7857 (time 1788157119.80 total 444.33 last 18.08) - open/close 8005 (time 1788157129.82 total 454.35 last 14.77) - open/close 8243 (time 1788157140.51 total 465.04 last 22.26) - open/close 8283 (time 1788157150.61 total 475.14 last 3.96) - open/close 8366 (time 1788157160.88 total 485.40 last 8.08) - open/close 8433 (time 1788157170.90 total 495.43 last 6.68) - open/close 8690 (time 1788157182.92 total 507.45 last 21.38) - open/close 8897 (time 1788157192.93 total 517.46 last 20.69) - open/close 9165 (time 1788157202.95 total 527.47 last 26.75) - open/close 9363 (time 1788157212.96 total 537.48 last 19.78) - open/close 9548 (time 1788157223.05 total 547.58 last 18.32) - open/close 9677 (time 1788157233.82 total 558.34 last 11.99) - open/close 9913 (time 1788157245.65 total 570.18 last 19.94) - open/close 10000 (time 1788157249.37 total 573.89 last 23.42) - open/close 10207 (time 1788157259.89 total 584.42 last 19.67) - open/close 10314 (time 1788157269.89 total 594.42 last 10.70) - open/close 10496 (time 1788157279.95 total 604.48 last 18.09) - open/close 10578 (time 1788157289.95 total 614.48 last 8.20) - open/close 10770 (time 1788157299.95 total 624.48 last 19.20) - open/close 10981 (time 1788157310.61 total 635.14 last 19.80) - open/close 11229 (time 1788157321.43 total 645.95 last 22.93) - open/close 11458 (time 1788157331.60 total 656.12 last 22.51) - open/close 11588 (time 1788157341.62 total 666.15 last 12.97) - open/close 11652 (time 1788157351.63 total 676.16 last 6.39) - open/close 11762 (time 1788157362.39 total 686.92 last 10.22) - open/close 12005 (time 1788157372.40 total 696.93 last 24.27) - open/close 12183 (time 1788157382.41 total 706.93 last 17.79) - open/close 12392 (time 1788157392.41 total 716.93 last 20.90) - open/close 12479 (time 1788157403.05 total 727.58 last 8.17) - open/close 12625 (time 1788157413.06 total 737.58 last 14.59) - open/close 12815 (time 1788157423.07 total 747.60 last 18.97) - open/close 13041 (time 1788157433.07 total 757.60 last 22.59) - open/close 13229 (time 1788157445.15 total 769.68 last 15.57) - open/close 13483 (time 1788157455.17 total 779.70 last 25.35) - open/close 13709 (time 1788157465.19 total 789.71 last 22.57) - open/close 13850 (time 1788157475.20 total 799.73 last 14.08) - open/close 14033 (time 1788157487.92 total 812.45 last 14.39) - open/close 14176 (time 1788157497.93 total 822.46 last 14.29) - open/close 14348 (time 1788157508.92 total 833.44 last 15.65) - open/close 14432 (time 1788157519.29 total 843.81 last 8.10) - open/close 14600 (time 1788157529.29 total 853.82 last 16.80) - open/close 14782 (time 1788157539.30 total 863.83 last 18.17) - open/close 15031 (time 1788157549.82 total 874.34 last 23.69) - open/close 15169 (time 1788157560.38 total 884.91 last 13.07) - open/close 15376 (time 1788157570.40 total 894.92 last 20.66) - open/close 15534 (time 1788157580.40 total 904.93 last 15.79) - open/close 15625 (time 1788157591.00 total 915.53 last 8.59) - open/close 15811 (time 1788157601.76 total 926.29 last 17.29) - open/close 16041 (time 1788157612.43 total 936.96 last 21.55) - open/close 16258 (time 1788157623.08 total 947.60 last 20.38) - open/close 16519 (time 1788157633.08 total 957.61 last 26.09) - open/close 16736 (time 1788157643.87 total 968.40 last 20.11) - open/close 16846 (time 1788157653.91 total 978.43 last 10.96) - open/close 17075 (time 1788157664.86 total 989.39 last 20.90) - open/close 17337 (time 1788157674.88 total 999.40 last 26.16) - open/close 17608 (time 1788157684.94 total 1009.46 last 26.94) - open/close 17977 (time 1788157694.94 total 1019.47 last 36.87) - open/close 18687 (time 1788157704.95 total 1029.47 last 70.98) - open/close 19428 (time 1788157714.96 total 1039.48 last 74.03) - open/close 20000 (time 1788157722.53 total 1047.06 last 75.54) - open/close 20720 (time 1788157732.53 total 1057.06 last 71.96) - open/close 21433 (time 1788157742.54 total 1067.07 last 71.26) - open/close 22150 (time 1788157752.54 total 1077.07 last 71.68) - open/close 22888 (time 1788157762.55 total 1087.08 last 73.74) - open/close 23483 (time 1788157772.55 total 1097.08 last 59.46) - open/close 24214 (time 1788157782.56 total 1107.09 last 73.07) - open/close 24914 (time 1788157792.58 total 1117.10 last 69.88) - open/close 25535 (time 1788157802.58 total 1127.10 last 62.09) - open/close 26134 (time 1788157812.59 total 1137.11 last 59.84) - open/close 26746 (time 1788157822.60 total 1147.13 last 61.11) - open/close 27356 (time 1788157832.60 total 1157.13 last 60.99) - open/close 28061 (time 1788157842.64 total 1167.17 last 70.24) - open/close 28711 (time 1788157852.64 total 1177.17 last 64.99) - open/close 29353 (time 1788157862.65 total 1187.17 last 64.16) - open/close 30000 (time 1788157871.64 total 1196.17 last 71.93) - open/close 30682 (time 1788157881.65 total 1206.18 last 68.15) - open/close 31401 (time 1788157891.65 total 1216.18 last 71.86) - open/close 32066 (time 1788157901.67 total 1226.20 last 66.41) - open/close 32800 (time 1788157911.67 total 1236.20 last 73.40) - open/close 33523 (time 1788157921.68 total 1246.20 last 72.24) - open/close 34290 (time 1788157931.69 total 1256.22 last 76.61) - open/close 35023 (time 1788157941.70 total 1266.22 last 73.24) - open/close 35780 (time 1788157951.71 total 1276.24 last 75.57) - open/close 36541 (time 1788157961.72 total 1286.24 last 76.09) - open/close 37271 (time 1788157971.72 total 1296.25 last 72.97) - open/close 37992 (time 1788157981.72 total 1306.25 last 72.08) - open/close 38724 (time 1788157991.74 total 1316.26 last 73.11) - open/close 39494 (time 1788158001.74 total 1326.27 last 76.95) - open/close 40000 (time 1788158008.95 total 1333.48 last 70.16) - open/close 40623 (time 1788158018.95 total 1343.48 last 62.29) - open/close 41329 (time 1788158028.97 total 1353.50 last 70.49) - open/close 42021 (time 1788158038.97 total 1363.50 last 69.18) - open/close 42742 (time 1788158048.98 total 1373.51 last 72.01) - open/close 43467 (time 1788158058.99 total 1383.52 last 72.45) - open/close 44161 (time 1788158069.00 total 1393.53 last 69.35) - open/close 44911 (time 1788158079.01 total 1403.54 last 74.91) - open/close 45731 (time 1788158089.01 total 1413.54 last 81.97) - open/close 46455 (time 1788158099.02 total 1423.55 last 72.33) - open/close 47122 (time 1788158109.03 total 1433.55 last 66.68) - open/close 47951 (time 1788158119.04 total 1443.57 last 82.75) - open/close 48610 (time 1788158129.05 total 1453.58 last 65.85) - open/close 49297 (time 1788158139.06 total 1463.59 last 68.65) total: 50000 open/close in 1472.77 seconds: 33.95 ops/second - unlinked 0 (time 1788158152 ; total 0 ; last 0) - unlinked 10000 (time 1788158291 ; total 139 ; last 139) - unlinked 20000 (time 1788158427 ; total 275 ; last 136) - unlinked 30000 (time 1788158571 ; total 419 ; last 144) - unlinked 40000 (time 1788158705 ; total 553 ; last 134) total: 50000 unlinks in 687 seconds: 72.780205 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg345-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg345-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg345-server stop mds service on oleg345-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg345-server unloading modules via unload_modules_local on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing unload_modules_local oleg345-server: modules unloaded. pdsh@oleg345-client: oleg345-client: ssh exited with exit code 2 pdsh@oleg345-client: oleg345-server: ssh exited with exit code 2 pdsh@oleg345-client: oleg345-client: ssh exited with exit code 2 pdsh@oleg345-client: oleg345-server: ssh exited with exit code 2