== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:49:26 (1789138166) start mds service on oleg639-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg639-server' oleg639-server: oleg639-server.virtnet: executing load_modules_local oleg639-server: Loading modules from /home/green/git/lustre-release/lustre oleg639-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg639-server: detected 4 online CPUs by sysfs oleg639-server: MODOPTS_LIBCFS= oleg639-server: Force libcfs to create 2 CPU partitions oleg639-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg639-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg639-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg639-server: oleg639-server.virtnet: executing set_default_debug -1 all pdsh@oleg639-client: oleg639-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg639-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg639-server: oleg639-server.virtnet: executing set_default_debug -1 all pdsh@oleg639-client: oleg639-server: ssh exited with exit code 1 Started lustre-OST0000 oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg639-client.virtnet: -o user_xattr,flock 192.168.206.139@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 254 (time 1789138238.25 total 10.00 last 25.39) - open/close 491 (time 1789138249.10 total 20.86 last 21.83) - open/close 732 (time 1789138259.10 total 30.86 last 24.09) - open/close 909 (time 1789138271.18 total 42.94 last 14.66) - open/close 1136 (time 1789138282.18 total 53.94 last 20.63) - open/close 1388 (time 1789138292.19 total 63.95 last 25.19) - open/close 1626 (time 1789138302.75 total 74.51 last 22.53) - open/close 1852 (time 1789138312.90 total 84.66 last 22.26) - open/close 2027 (time 1789138323.40 total 95.16 last 16.67) - open/close 2195 (time 1789138334.27 total 106.03 last 15.45) - open/close 2415 (time 1789138344.39 total 116.15 last 21.74) - open/close 2643 (time 1789138354.40 total 126.15 last 22.79) - open/close 2889 (time 1789138364.55 total 136.30 last 24.24) - open/close 3039 (time 1789138374.55 total 146.31 last 14.99) - open/close 3227 (time 1789138384.84 total 156.60 last 18.27) - open/close 3433 (time 1789138395.34 total 167.10 last 19.61) - open/close 3657 (time 1789138406.36 total 178.11 last 20.34) - open/close 3892 (time 1789138416.91 total 188.67 last 22.26) - open/close 4098 (time 1789138427.92 total 199.68 last 18.72) - open/close 4318 (time 1789138437.92 total 209.68 last 21.98) - open/close 4538 (time 1789138447.93 total 219.69 last 21.98) - open/close 4736 (time 1789138457.94 total 229.70 last 19.78) - open/close 4975 (time 1789138467.95 total 239.71 last 23.89) - open/close 5197 (time 1789138478.79 total 250.55 last 20.48) - open/close 5446 (time 1789138489.10 total 260.86 last 24.14) - open/close 5685 (time 1789138499.73 total 271.49 last 22.49) - open/close 5881 (time 1789138509.77 total 281.52 last 19.52) - open/close 6028 (time 1789138519.79 total 291.55 last 14.66) - open/close 6286 (time 1789138532.30 total 304.06 last 20.63) - open/close 6564 (time 1789138542.70 total 314.46 last 26.72) - open/close 6826 (time 1789138553.68 total 325.44 last 23.87) - open/close 7077 (time 1789138563.70 total 335.45 last 25.06) - open/close 7277 (time 1789138573.73 total 345.49 last 19.93) - open/close 7569 (time 1789138584.72 total 356.47 last 26.59) - open/close 7754 (time 1789138594.76 total 366.52 last 18.42) - open/close 7919 (time 1789138605.39 total 377.15 last 15.52) - open/close 8008 (time 1789138615.39 total 387.15 last 8.90) - open/close 8187 (time 1789138625.41 total 397.16 last 17.88) - open/close 8387 (time 1789138636.20 total 407.96 last 18.53) - open/close 8576 (time 1789138646.20 total 417.96 last 18.90) - open/close 8818 (time 1789138656.69 total 428.45 last 23.07) - open/close 9091 (time 1789138666.96 total 438.72 last 26.59) - open/close 9379 (time 1789138677.71 total 449.47 last 26.79) - open/close 9614 (time 1789138687.73 total 459.49 last 23.45) - open/close 9861 (time 1789138698.39 total 470.15 last 23.18) - open/close 10000 (time 1789138703.39 total 475.15 last 27.79) - open/close 10256 (time 1789138713.76 total 485.52 last 24.68) - open/close 10537 (time 1789138724.81 total 496.57 last 25.43) - open/close 10847 (time 1789138735.82 total 507.58 last 28.16) - open/close 11092 (time 1789138746.06 total 517.81 last 23.95) - open/close 11278 (time 1789138756.76 total 528.52 last 17.37) - open/close 11523 (time 1789138766.77 total 538.53 last 24.47) - open/close 11690 (time 1789138776.79 total 548.54 last 16.68) - open/close 11756 (time 1789138786.79 total 558.55 last 6.60) - open/close 11881 (time 1789138796.82 total 568.58 last 12.47) - open/close 11995 (time 1789138807.31 total 579.07 last 10.86) - open/close 12116 (time 1789138817.33 total 589.09 last 12.08) - open/close 12290 (time 1789138828.14 total 599.90 last 16.09) - open/close 12518 (time 1789138838.17 total 609.93 last 22.74) - open/close 12698 (time 1789138848.18 total 619.93 last 17.99) - open/close 12853 (time 1789138858.18 total 629.94 last 15.49) - open/close 13053 (time 1789138868.48 total 640.24 last 19.42) - open/close 13236 (time 1789138879.43 total 651.19 last 16.72) - open/close 13486 (time 1789138889.43 total 661.19 last 25.00) - open/close 13655 (time 1789138899.61 total 671.36 last 16.61) - open/close 13779 (time 1789138910.35 total 682.11 last 11.54) - open/close 13995 (time 1789138921.17 total 692.92 last 19.98) - open/close 14205 (time 1789138931.34 total 703.10 last 20.64) - open/close 14458 (time 1789138942.19 total 713.94 last 23.32) - open/close 14697 (time 1789138952.21 total 723.96 last 23.86) - open/close 14898 (time 1789138962.33 total 734.09 last 19.85) - open/close 15124 (time 1789138972.56 total 744.32 last 22.09) - open/close 15362 (time 1789138983.37 total 755.13 last 22.02) - open/close 15562 (time 1789138993.38 total 765.14 last 19.98) - open/close 15760 (time 1789139003.40 total 775.16 last 19.75) - open/close 15987 (time 1789139013.41 total 785.17 last 22.69) - open/close 16263 (time 1789139024.26 total 796.02 last 25.43) - open/close 16540 (time 1789139035.17 total 806.93 last 25.40) - open/close 16768 (time 1789139045.17 total 816.93 last 22.79) - open/close 16983 (time 1789139056.67 total 828.43 last 18.69) - open/close 17194 (time 1789139067.40 total 839.16 last 19.68) - open/close 17486 (time 1789139077.52 total 849.27 last 28.86) - open/close 17770 (time 1789139087.52 total 859.27 last 28.40) - open/close 18001 (time 1789139097.53 total 869.29 last 23.07) - open/close 18257 (time 1789139107.59 total 879.35 last 25.43) - open/close 18490 (time 1789139117.71 total 889.47 last 23.03) - open/close 18776 (time 1789139128.52 total 900.28 last 26.46) - open/close 19019 (time 1789139139.09 total 910.85 last 22.99) - open/close 19614 (time 1789139149.10 total 920.86 last 59.45) - open/close 20000 (time 1789139154.19 total 925.95 last 75.81) - open/close 20805 (time 1789139164.20 total 935.96 last 80.41) - open/close 21501 (time 1789139174.20 total 945.96 last 69.58) - open/close 22336 (time 1789139184.27 total 956.02 last 82.99) - open/close 23084 (time 1789139194.27 total 966.03 last 74.76) - open/close 23924 (time 1789139204.28 total 976.04 last 83.89) - open/close 24749 (time 1789139214.30 total 986.06 last 82.35) - open/close 25486 (time 1789139224.30 total 996.06 last 73.69) - open/close 26229 (time 1789139234.30 total 1006.06 last 74.30) - open/close 26901 (time 1789139244.32 total 1016.07 last 67.12) - open/close 27703 (time 1789139254.33 total 1026.09 last 80.06) - open/close 28508 (time 1789139264.34 total 1036.10 last 80.41) - open/close 29378 (time 1789139274.35 total 1046.10 last 86.98) - open/close 30000 (time 1789139281.64 total 1053.40 last 85.23) - open/close 30813 (time 1789139291.65 total 1063.41 last 81.21) - open/close 31540 (time 1789139301.68 total 1073.43 last 72.54) - open/close 32353 (time 1789139311.69 total 1083.45 last 81.20) - open/close 33139 (time 1789139321.71 total 1093.47 last 78.43) - open/close 33919 (time 1789139331.71 total 1103.47 last 77.98) - open/close 34727 (time 1789139341.73 total 1113.48 last 80.70) - open/close 35452 (time 1789139351.73 total 1123.49 last 72.48) - open/close 36348 (time 1789139361.73 total 1133.49 last 89.58) - open/close 37046 (time 1789139371.73 total 1143.49 last 69.78) - open/close 37874 (time 1789139381.74 total 1153.50 last 82.75) - open/close 38673 (time 1789139391.74 total 1163.50 last 79.87) - open/close 39412 (time 1789139401.75 total 1173.50 last 73.87) - open/close 40000 (time 1789139411.26 total 1183.01 last 61.84) - open/close 40816 (time 1789139421.26 total 1193.02 last 81.52) - open/close 41513 (time 1789139431.28 total 1203.03 last 69.63) - open/close 42266 (time 1789139441.28 total 1213.04 last 75.27) - open/close 43072 (time 1789139451.29 total 1223.05 last 80.48) - open/close 43886 (time 1789139461.30 total 1233.06 last 81.36) - open/close 44746 (time 1789139471.30 total 1243.06 last 85.97) - open/close 45569 (time 1789139481.31 total 1253.07 last 82.19) - open/close 46426 (time 1789139491.33 total 1263.08 last 85.61) - open/close 47219 (time 1789139501.33 total 1273.09 last 79.27) - open/close 48045 (time 1789139511.34 total 1283.10 last 82.52) - open/close 48806 (time 1789139521.34 total 1293.10 last 76.08) - open/close 49637 (time 1789139531.36 total 1303.11 last 82.99) total: 50000 open/close in 1307.56 seconds: 38.24 ops/second - unlinked 0 (time 1789139539 ; total 0 ; last 0) - unlinked 10000 (time 1789139667 ; total 128 ; last 128) - unlinked 20000 (time 1789139799 ; total 260 ; last 132) - unlinked 30000 (time 1789139924 ; total 385 ; last 125) - unlinked 40000 (time 1789140055 ; total 516 ; last 131) total: 50000 unlinks in 651 seconds: 76.804916 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg639-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg639-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg639-server stop mds service on oleg639-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg639-server unloading modules via unload_modules_local on: 'oleg639-server' oleg639-server: oleg639-server.virtnet: executing unload_modules_local oleg639-server: modules unloaded. pdsh@oleg639-client: oleg639-client: ssh exited with exit code 2 pdsh@oleg639-client: oleg639-server: ssh exited with exit code 2 pdsh@oleg639-client: oleg639-client: ssh exited with exit code 2 pdsh@oleg639-client: oleg639-server: ssh exited with exit code 2