== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 15:39:17 (1783971557) start mds service on oleg341-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg341-server' oleg341-server: oleg341-server.virtnet: executing load_modules_local oleg341-server: Loading modules from /home/green/git/lustre-release/lustre oleg341-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg341-server: detected 4 online CPUs by sysfs oleg341-server: MODOPTS_LIBCFS= oleg341-server: Force libcfs to create 2 CPU partitions oleg341-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg341-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg341-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg341-server: oleg341-server.virtnet: executing set_default_debug -1 all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg341-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg341-server: oleg341-server.virtnet: executing set_default_debug -1 all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Started lustre-OST0000 oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg341-client.virtnet: -o user_xattr,flock 192.168.203.141@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 234 (time 1783971642.64 total 10.10 last 23.18) - open/close 461 (time 1783971652.64 total 20.10 last 22.70) - open/close 721 (time 1783971662.65 total 30.11 last 25.96) - open/close 914 (time 1783971672.65 total 40.12 last 19.29) - open/close 1065 (time 1783971682.77 total 50.23 last 14.93) - open/close 1295 (time 1783971693.38 total 60.84 last 21.67) - open/close 1522 (time 1783971704.15 total 71.61 last 21.08) - open/close 1773 (time 1783971714.18 total 81.64 last 25.03) - open/close 2029 (time 1783971724.61 total 92.07 last 24.54) - open/close 2233 (time 1783971735.50 total 102.96 last 18.73) - open/close 2429 (time 1783971745.62 total 113.08 last 19.37) - open/close 2586 (time 1783971756.32 total 123.78 last 14.68) - open/close 2737 (time 1783971767.45 total 134.91 last 13.56) - open/close 2957 (time 1783971777.46 total 144.92 last 21.99) - open/close 3144 (time 1783971790.17 total 157.63 last 14.71) - open/close 3359 (time 1783971800.26 total 167.72 last 21.30) - open/close 3642 (time 1783971810.28 total 177.74 last 28.26) - open/close 3849 (time 1783971820.28 total 187.74 last 20.69) - open/close 4125 (time 1783971830.93 total 198.39 last 25.91) - open/close 4340 (time 1783971840.98 total 208.44 last 21.40) - open/close 4567 (time 1783971851.23 total 218.69 last 22.14) - open/close 4798 (time 1783971861.51 total 228.97 last 22.46) - open/close 5003 (time 1783971871.63 total 239.09 last 20.26) - open/close 5163 (time 1783971883.01 total 250.47 last 14.06) - open/close 5228 (time 1783971895.89 total 263.35 last 5.05) - open/close 5449 (time 1783971906.45 total 273.91 last 20.93) - open/close 5663 (time 1783971916.69 total 284.15 last 20.90) - open/close 5941 (time 1783971927.69 total 295.15 last 25.28) - open/close 6144 (time 1783971937.90 total 305.36 last 19.88) - open/close 6356 (time 1783971948.74 total 316.20 last 19.57) - open/close 6570 (time 1783971958.75 total 326.21 last 21.38) - open/close 6842 (time 1783971968.85 total 336.31 last 26.93) - open/close 7060 (time 1783971978.96 total 346.42 last 21.55) - open/close 7280 (time 1783971988.97 total 356.43 last 21.98) - open/close 7439 (time 1783971999.00 total 366.46 last 15.85) - open/close 7650 (time 1783972009.00 total 376.47 last 21.10) - open/close 7873 (time 1783972019.27 total 386.73 last 21.73) - open/close 8079 (time 1783972029.76 total 397.22 last 19.62) - open/close 8301 (time 1783972039.77 total 407.23 last 22.19) - open/close 8507 (time 1783972050.83 total 418.29 last 18.62) - open/close 8747 (time 1783972061.06 total 428.52 last 23.46) - open/close 8964 (time 1783972071.07 total 438.53 last 21.68) - open/close 9166 (time 1783972081.64 total 449.10 last 19.11) - open/close 9364 (time 1783972093.99 total 461.45 last 16.02) - open/close 9619 (time 1783972104.01 total 471.47 last 25.46) - open/close 9816 (time 1783972114.79 total 482.25 last 18.27) - open/close 9980 (time 1783972125.00 total 492.47 last 16.06) - open/close 10140 (time 1783972135.80 total 503.26 last 14.83) - open/close 10336 (time 1783972145.80 total 513.26 last 19.60) - open/close 10472 (time 1783972156.17 total 523.63 last 13.11) - open/close 10698 (time 1783972166.18 total 533.64 last 22.58) - open/close 10922 (time 1783972177.16 total 544.63 last 20.39) - open/close 11101 (time 1783972187.17 total 554.63 last 17.90) - open/close 11177 (time 1783972197.83 total 565.29 last 7.13) - open/close 11416 (time 1783972207.84 total 575.30 last 23.89) - open/close 11654 (time 1783972218.31 total 585.77 last 22.71) - open/close 11968 (time 1783972228.64 total 596.10 last 30.41) - open/close 12198 (time 1783972239.24 total 606.70 last 21.71) - open/close 12333 (time 1783972249.24 total 616.70 last 13.49) - open/close 12600 (time 1783972260.25 total 627.71 last 24.26) - open/close 12749 (time 1783972270.54 total 638.00 last 14.47) - open/close 12970 (time 1783972280.84 total 648.30 last 21.47) - open/close 13186 (time 1783972290.89 total 658.35 last 21.49) - open/close 13396 (time 1783972301.60 total 669.06 last 19.60) - open/close 13648 (time 1783972312.60 total 680.06 last 22.92) - open/close 13868 (time 1783972322.78 total 690.24 last 21.61) - open/close 14085 (time 1783972335.48 total 702.94 last 17.09) - open/close 14212 (time 1783972345.50 total 712.96 last 12.68) - open/close 14423 (time 1783972355.53 total 722.99 last 21.03) - open/close 14609 (time 1783972365.56 total 733.02 last 18.55) - open/close 14827 (time 1783972375.58 total 743.04 last 21.76) - open/close 14895 (time 1783972386.51 total 753.97 last 6.22) - open/close 15117 (time 1783972397.03 total 764.49 last 21.10) - open/close 15303 (time 1783972407.83 total 775.29 last 17.22) - open/close 15544 (time 1783972418.31 total 785.77 last 23.00) - open/close 15783 (time 1783972428.89 total 796.35 last 22.60) - open/close 15960 (time 1783972439.64 total 807.10 last 16.46) - open/close 16181 (time 1783972450.73 total 818.19 last 19.93) - open/close 16357 (time 1783972460.73 total 828.19 last 17.59) - open/close 16551 (time 1783972470.74 total 838.20 last 19.39) - open/close 16760 (time 1783972481.70 total 849.16 last 19.06) - open/close 16993 (time 1783972491.71 total 859.17 last 23.28) - open/close 17149 (time 1783972502.23 total 869.69 last 14.83) - open/close 17250 (time 1783972512.78 total 880.25 last 9.57) - open/close 17428 (time 1783972523.09 total 890.55 last 17.27) - open/close 17599 (time 1783972533.33 total 900.79 last 16.70) - open/close 17685 (time 1783972543.34 total 910.80 last 8.59) - open/close 18031 (time 1783972553.34 total 920.80 last 34.59) - open/close 18653 (time 1783972563.35 total 930.81 last 62.14) - open/close 19287 (time 1783972573.36 total 940.82 last 63.34) - open/close 19960 (time 1783972583.37 total 950.83 last 67.26) - open/close 20622 (time 1783972593.37 total 960.83 last 66.17) - open/close 21309 (time 1783972603.37 total 970.83 last 68.70) - open/close 21957 (time 1783972613.38 total 980.84 last 64.72) - open/close 22594 (time 1783972623.39 total 990.85 last 63.65) - open/close 23160 (time 1783972633.39 total 1000.86 last 56.58) - open/close 23853 (time 1783972643.40 total 1010.86 last 69.24) - open/close 24525 (time 1783972653.42 total 1020.88 last 67.11) - open/close 25185 (time 1783972663.43 total 1030.89 last 65.94) - open/close 25859 (time 1783972673.43 total 1040.90 last 67.34) - open/close 26568 (time 1783972683.46 total 1050.92 last 70.75) - open/close 27211 (time 1783972693.49 total 1060.95 last 64.11) - open/close 27884 (time 1783972703.49 total 1070.95 last 67.26) - open/close 28525 (time 1783972713.50 total 1080.96 last 64.04) - open/close 29159 (time 1783972723.51 total 1090.97 last 63.35) - open/close 29822 (time 1783972733.51 total 1100.97 last 66.29) - open/close 30000 (time 1783972735.96 total 1103.42 last 72.85) - open/close 30637 (time 1783972745.96 total 1113.42 last 63.65) - open/close 31316 (time 1783972755.98 total 1123.44 last 67.81) - open/close 31930 (time 1783972765.99 total 1133.45 last 61.32) - open/close 32568 (time 1783972776.00 total 1143.46 last 63.74) - open/close 33260 (time 1783972786.00 total 1153.47 last 69.16) - open/close 33924 (time 1783972796.01 total 1163.47 last 66.36) - open/close 34554 (time 1783972806.01 total 1173.47 last 62.99) - open/close 35097 (time 1783972816.01 total 1183.48 last 54.28) - open/close 35684 (time 1783972826.03 total 1193.49 last 58.64) - open/close 36324 (time 1783972836.04 total 1203.50 last 63.91) - open/close 36929 (time 1783972846.05 total 1213.51 last 60.41) - open/close 37562 (time 1783972856.06 total 1223.52 last 63.25) - open/close 38111 (time 1783972866.08 total 1233.54 last 54.82) - open/close 38749 (time 1783972876.08 total 1243.54 last 63.77) - open/close 39406 (time 1783972886.09 total 1253.56 last 65.61) - open/close 40000 (time 1783972895.66 total 1263.12 last 62.12) - open/close 40552 (time 1783972905.68 total 1273.14 last 55.08) - open/close 41136 (time 1783972915.70 total 1283.16 last 58.26) - open/close 41721 (time 1783972925.71 total 1293.17 last 58.45) - open/close 42316 (time 1783972935.73 total 1303.19 last 59.39) - open/close 42929 (time 1783972945.73 total 1313.20 last 61.27) - open/close 43613 (time 1783972955.74 total 1323.20 last 68.36) - open/close 44268 (time 1783972965.74 total 1333.20 last 65.47) - open/close 44845 (time 1783972975.75 total 1343.21 last 57.69) - open/close 45378 (time 1783972985.75 total 1353.21 last 53.28) - open/close 45981 (time 1783972995.76 total 1363.22 last 60.24) - open/close 46559 (time 1783973005.76 total 1373.22 last 57.78) - open/close 47164 (time 1783973015.77 total 1383.23 last 60.44) - open/close 47806 (time 1783973025.80 total 1393.26 last 64.05) - open/close 48499 (time 1783973035.80 total 1403.26 last 69.30) - open/close 49163 (time 1783973045.80 total 1413.26 last 66.36) - open/close 49813 (time 1783973055.81 total 1423.27 last 64.96) total: 50000 open/close in 1426.33 seconds: 35.06 ops/second - unlinked 0 (time 1783973063 ; total 1 ; last 1) - unlinked 10000 (time 1783973234 ; total 172 ; last 171) - unlinked 20000 (time 1783973408 ; total 346 ; last 174) - unlinked 30000 (time 1783973568 ; total 506 ; last 160) - unlinked 40000 (time 1783973730 ; total 668 ; last 162) total: 50000 unlinks in 832 seconds: 60.096153 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg341-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg341-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg341-server stop mds service on oleg341-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg341-server unloading modules via unload_modules_local on: 'oleg341-server' oleg341-server: oleg341-server.virtnet: executing unload_modules_local oleg341-server: modules unloaded. pdsh@oleg341-client: oleg341-client: ssh exited with exit code 2 pdsh@oleg341-client: oleg341-server: ssh exited with exit code 2 pdsh@oleg341-client: oleg341-client: ssh exited with exit code 2 pdsh@oleg341-client: oleg341-server: ssh exited with exit code 2