== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:07:02 (1788444422) start mds service on oleg210-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg210-server' oleg210-server: oleg210-server.virtnet: executing load_modules_local oleg210-server: Loading modules from /home/green/git/lustre-release/lustre oleg210-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg210-server: detected 4 online CPUs by sysfs oleg210-server: MODOPTS_LIBCFS= oleg210-server: Force libcfs to create 2 CPU partitions oleg210-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg210-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg210-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg210-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 203 (time 1788444506.41 total 10.02 last 20.26) - open/close 476 (time 1788444516.76 total 20.37 last 26.38) - open/close 687 (time 1788444527.72 total 31.33 last 19.25) - open/close 983 (time 1788444538.28 total 41.89 last 28.03) - open/close 1204 (time 1788444548.92 total 52.53 last 20.77) - open/close 1476 (time 1788444558.93 total 62.54 last 27.17) - open/close 1689 (time 1788444568.96 total 72.57 last 21.23) - open/close 1941 (time 1788444579.70 total 83.31 last 23.48) - open/close 2161 (time 1788444590.00 total 93.61 last 21.36) - open/close 2313 (time 1788444600.00 total 103.61 last 15.20) - open/close 2532 (time 1788444610.40 total 114.01 last 21.05) - open/close 2777 (time 1788444620.82 total 124.43 last 23.51) - open/close 3062 (time 1788444630.84 total 134.45 last 28.46) - open/close 3301 (time 1788444641.06 total 144.67 last 23.38) - open/close 3553 (time 1788444651.84 total 155.45 last 23.38) - open/close 3845 (time 1788444661.84 total 165.45 last 29.19) - open/close 4011 (time 1788444672.29 total 175.90 last 15.88) - open/close 4223 (time 1788444682.76 total 186.37 last 20.25) - open/close 4405 (time 1788444692.76 total 196.37 last 18.20) - open/close 4486 (time 1788444702.77 total 206.38 last 8.09) - open/close 4709 (time 1788444713.52 total 217.13 last 20.75) - open/close 5065 (time 1788444723.54 total 227.15 last 35.55) - open/close 5290 (time 1788444734.10 total 237.71 last 21.30) - open/close 5532 (time 1788444744.11 total 247.72 last 24.18) - open/close 5801 (time 1788444755.18 total 258.79 last 24.29) - open/close 5990 (time 1788444766.03 total 269.64 last 17.42) - open/close 6167 (time 1788444776.03 total 279.64 last 17.69) - open/close 6358 (time 1788444786.87 total 290.48 last 17.62) - open/close 6521 (time 1788444797.15 total 300.77 last 15.85) - open/close 6797 (time 1788444807.18 total 310.79 last 27.52) - open/close 7024 (time 1788444817.60 total 321.22 last 21.78) - open/close 7258 (time 1788444827.63 total 331.24 last 23.34) - open/close 7351 (time 1788444838.04 total 341.65 last 8.94) - open/close 7514 (time 1788444848.99 total 352.60 last 14.88) - open/close 7727 (time 1788444859.44 total 363.05 last 20.39) - open/close 7908 (time 1788444869.44 total 373.05 last 18.10) - open/close 8106 (time 1788444879.44 total 383.05 last 19.79) - open/close 8358 (time 1788444890.34 total 393.95 last 23.13) - open/close 8595 (time 1788444900.35 total 403.96 last 23.68) - open/close 8781 (time 1788444910.35 total 413.96 last 18.59) - open/close 8962 (time 1788444920.48 total 424.09 last 17.87) - open/close 9152 (time 1788444930.49 total 434.10 last 18.97) - open/close 9273 (time 1788444943.19 total 446.80 last 9.53) - open/close 9489 (time 1788444953.19 total 456.80 last 21.60) - open/close 9658 (time 1788444963.76 total 467.38 last 15.98) - open/close 9795 (time 1788444974.62 total 478.24 last 12.62) - open/close 9988 (time 1788444984.68 total 488.29 last 19.20) - open/close 10248 (time 1788444994.70 total 498.32 last 25.93) - open/close 10431 (time 1788445004.95 total 508.57 last 17.86) - open/close 10604 (time 1788445014.95 total 518.57 last 17.30) - open/close 10765 (time 1788445024.97 total 528.58 last 16.07) - open/close 10885 (time 1788445035.12 total 538.73 last 11.83) - open/close 11052 (time 1788445045.60 total 549.21 last 15.92) - open/close 11245 (time 1788445055.72 total 559.33 last 19.07) - open/close 11419 (time 1788445066.01 total 569.62 last 16.92) - open/close 11560 (time 1788445076.11 total 579.73 last 13.95) - open/close 11757 (time 1788445086.56 total 590.17 last 18.87) - open/close 11950 (time 1788445096.75 total 600.36 last 18.94) - open/close 12182 (time 1788445106.79 total 610.40 last 23.09) - open/close 12433 (time 1788445117.55 total 621.16 last 23.34) - open/close 12634 (time 1788445128.24 total 631.85 last 18.79) - open/close 12846 (time 1788445140.80 total 644.41 last 16.88) - open/close 12972 (time 1788445153.26 total 656.87 last 10.11) - open/close 13145 (time 1788445163.27 total 666.88 last 17.29) - open/close 13360 (time 1788445173.27 total 676.89 last 21.49) - open/close 13547 (time 1788445183.99 total 687.60 last 17.45) - open/close 13780 (time 1788445195.86 total 699.48 last 19.63) - open/close 14020 (time 1788445206.49 total 710.10 last 22.59) - open/close 14217 (time 1788445216.60 total 720.21 last 19.48) - open/close 14477 (time 1788445226.79 total 730.40 last 25.52) - open/close 14640 (time 1788445236.81 total 740.42 last 16.27) - open/close 14828 (time 1788445247.13 total 750.74 last 18.21) - open/close 15002 (time 1788445258.15 total 761.76 last 15.80) - open/close 15073 (time 1788445268.64 total 772.25 last 6.76) - open/close 15233 (time 1788445278.65 total 782.26 last 15.98) - open/close 15453 (time 1788445288.99 total 792.60 last 21.29) - open/close 15685 (time 1788445299.86 total 803.47 last 21.34) - open/close 15852 (time 1788445309.87 total 813.48 last 16.68) - open/close 16097 (time 1788445319.88 total 823.49 last 24.49) - open/close 16226 (time 1788445329.88 total 833.49 last 12.90) - open/close 16456 (time 1788445340.53 total 844.15 last 21.58) - open/close 16682 (time 1788445351.08 total 854.69 last 21.43) - open/close 16883 (time 1788445361.10 total 864.71 last 20.06) - open/close 16972 (time 1788445371.12 total 874.74 last 8.88) - open/close 17134 (time 1788445381.13 total 884.74 last 16.19) - open/close 17348 (time 1788445391.13 total 894.74 last 21.40) - open/close 17514 (time 1788445401.84 total 905.45 last 15.51) - open/close 17771 (time 1788445412.65 total 916.26 last 23.76) - open/close 17907 (time 1788445422.74 total 926.36 last 13.47) - open/close 18090 (time 1788445435.35 total 938.96 last 14.52) - open/close 18247 (time 1788445446.16 total 949.78 last 14.52) - open/close 18371 (time 1788445456.18 total 959.79 last 12.38) - open/close 18523 (time 1788445466.21 total 969.82 last 15.16) - open/close 18783 (time 1788445476.21 total 979.82 last 25.99) - open/close 19125 (time 1788445486.21 total 989.82 last 34.19) - open/close 20000 (time 1788445496.02 total 999.63 last 89.20) - open/close 20745 (time 1788445506.04 total 1009.65 last 74.33) - open/close 21498 (time 1788445516.05 total 1019.66 last 75.26) - open/close 22256 (time 1788445526.06 total 1029.67 last 75.72) - open/close 23117 (time 1788445536.07 total 1039.68 last 86.03) - open/close 23888 (time 1788445546.08 total 1049.69 last 77.01) - open/close 24790 (time 1788445556.08 total 1059.69 last 90.19) - open/close 25596 (time 1788445566.08 total 1069.70 last 80.56) - open/close 26354 (time 1788445576.10 total 1079.71 last 75.72) - open/close 27084 (time 1788445586.10 total 1089.71 last 72.95) - open/close 27822 (time 1788445596.11 total 1099.73 last 73.71) - open/close 28512 (time 1788445606.13 total 1109.74 last 68.92) - open/close 29264 (time 1788445616.13 total 1119.74 last 75.19) - open/close 30000 (time 1788445625.89 total 1129.50 last 75.38) - open/close 30835 (time 1788445635.89 total 1139.50 last 83.48) - open/close 31584 (time 1788445645.90 total 1149.51 last 74.82) - open/close 32404 (time 1788445655.91 total 1159.52 last 81.97) - open/close 33182 (time 1788445665.93 total 1169.54 last 77.63) - open/close 34033 (time 1788445675.93 total 1179.55 last 85.06) - open/close 34948 (time 1788445685.94 total 1189.55 last 91.43) - open/close 35714 (time 1788445695.95 total 1199.56 last 76.57) - open/close 36531 (time 1788445705.97 total 1209.58 last 81.47) - open/close 37353 (time 1788445715.98 total 1219.59 last 82.15) - open/close 38046 (time 1788445725.99 total 1229.60 last 69.25) - open/close 38847 (time 1788445736.00 total 1239.61 last 80.00) - open/close 39654 (time 1788445746.01 total 1249.62 last 80.63) - open/close 40000 (time 1788445750.07 total 1253.68 last 85.22) - open/close 40721 (time 1788445760.08 total 1263.69 last 72.03) - open/close 41481 (time 1788445770.09 total 1273.70 last 75.89) - open/close 42220 (time 1788445780.10 total 1283.71 last 73.85) - open/close 42839 (time 1788445790.11 total 1293.72 last 61.82) - open/close 43477 (time 1788445800.14 total 1303.76 last 63.59) - open/close 44020 (time 1788445810.16 total 1313.78 last 54.19) - open/close 44657 (time 1788445820.17 total 1323.78 last 63.65) - open/close 45390 (time 1788445830.17 total 1333.79 last 73.28) - open/close 46236 (time 1788445840.18 total 1343.79 last 84.55) - open/close 47041 (time 1788445850.19 total 1353.80 last 80.42) - open/close 47827 (time 1788445860.20 total 1363.81 last 78.55) - open/close 48659 (time 1788445870.20 total 1373.81 last 83.19) - open/close 49460 (time 1788445880.20 total 1383.81 last 80.07) total: 50000 open/close in 1390.49 seconds: 35.96 ops/second - unlinked 0 (time 1788445891 ; total 0 ; last 0) - unlinked 10000 (time 1788446029 ; total 138 ; last 138) - unlinked 20000 (time 1788446160 ; total 269 ; last 131) - unlinked 30000 (time 1788446287 ; total 396 ; last 127) - unlinked 40000 (time 1788446426 ; total 535 ; last 139) total: 50000 unlinks in 668 seconds: 74.850296 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg210-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server stop mds service on oleg210-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg210-server unloading modules via unload_modules_local on: 'oleg210-server' oleg210-server: oleg210-server.virtnet: executing unload_modules_local oleg210-server: modules unloaded. pdsh@oleg210-client: oleg210-client: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-server: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-client: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-server: ssh exited with exit code 2