== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 12:52:26 (1780332746) start mds service on oleg132-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg132-server' oleg132-server: oleg132-server.virtnet: executing load_modules_local oleg132-server: Loading modules from /home/green/git/lustre-release/lustre oleg132-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg132-server: detected 4 online CPUs by sysfs oleg132-server: MODOPTS_LIBCFS= oleg132-server: Force libcfs to create 2 CPU partitions oleg132-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg132-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg132-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg132-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-OST0000 oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 115 (time 1780332827.46 total 10.23 last 11.24) - open/close 234 (time 1780332837.46 total 20.23 last 11.89) - open/close 405 (time 1780332850.18 total 32.95 last 13.45) - open/close 541 (time 1780332863.10 total 45.87 last 10.53) - open/close 666 (time 1780332874.29 total 57.06 last 11.17) - open/close 741 (time 1780332884.53 total 67.31 last 7.32) - open/close 804 (time 1780332895.60 total 78.37 last 5.69) - open/close 910 (time 1780332906.29 total 89.06 last 9.92) - open/close 1091 (time 1780332916.61 total 99.38 last 17.55) - open/close 1350 (time 1780332926.62 total 109.39 last 25.88) - open/close 1556 (time 1780332937.64 total 120.41 last 18.68) - open/close 1636 (time 1780332948.42 total 131.19 last 7.43) - open/close 1731 (time 1780332959.38 total 142.15 last 8.66) - open/close 1866 (time 1780332969.39 total 152.16 last 13.48) - open/close 1972 (time 1780332979.40 total 162.18 last 10.59) - open/close 2102 (time 1780332989.81 total 172.58 last 12.50) - open/close 2252 (time 1780333000.07 total 182.84 last 14.62) - open/close 2410 (time 1780333010.48 total 193.25 last 15.17) - open/close 2540 (time 1780333020.99 total 203.76 last 12.37) - open/close 2752 (time 1780333031.00 total 213.77 last 21.18) - open/close 2814 (time 1780333041.71 total 224.48 last 5.79) - open/close 2939 (time 1780333052.01 total 234.78 last 12.13) - open/close 3056 (time 1780333062.89 total 245.66 last 10.75) - open/close 3262 (time 1780333072.93 total 255.71 last 20.51) - open/close 3371 (time 1780333082.96 total 265.73 last 10.88) - open/close 3402 (time 1780333093.06 total 275.84 last 3.07) - open/close 3518 (time 1780333103.43 total 286.20 last 11.19) - open/close 3660 (time 1780333113.45 total 296.22 last 14.18) - open/close 3719 (time 1780333123.47 total 306.24 last 5.89) - open/close 3830 (time 1780333133.48 total 316.25 last 11.09) - open/close 3936 (time 1780333144.38 total 327.15 last 9.72) - open/close 4114 (time 1780333155.07 total 337.84 last 16.65) - open/close 4272 (time 1780333165.09 total 347.86 last 15.76) - open/close 4466 (time 1780333176.05 total 358.82 last 17.71) - open/close 4574 (time 1780333186.06 total 368.84 last 10.78) - open/close 4746 (time 1780333196.08 total 378.85 last 17.18) - open/close 4875 (time 1780333206.18 total 388.95 last 12.76) - open/close 5051 (time 1780333216.19 total 398.96 last 17.59) - open/close 5218 (time 1780333226.21 total 408.98 last 16.67) - open/close 5278 (time 1780333236.22 total 418.99 last 5.99) - open/close 5345 (time 1780333246.96 total 429.73 last 6.24) - open/close 5510 (time 1780333257.45 total 440.22 last 15.74) - open/close 5656 (time 1780333268.34 total 451.11 last 13.40) - open/close 5775 (time 1780333278.70 total 461.48 last 11.48) - open/close 5870 (time 1780333289.58 total 472.35 last 8.74) - open/close 5984 (time 1780333299.58 total 482.35 last 11.39) - open/close 6149 (time 1780333309.61 total 492.39 last 16.45) - open/close 6221 (time 1780333320.19 total 502.96 last 6.81) - open/close 6343 (time 1780333330.66 total 513.43 last 11.65) - open/close 6422 (time 1780333341.24 total 524.01 last 7.46) - open/close 6560 (time 1780333352.31 total 535.08 last 12.47) - open/close 6738 (time 1780333362.32 total 545.09 last 17.78) - open/close 6827 (time 1780333372.54 total 555.31 last 8.71) - open/close 6935 (time 1780333382.55 total 565.32 last 10.80) - open/close 7095 (time 1780333393.52 total 576.29 last 14.58) - open/close 7182 (time 1780333403.96 total 586.73 last 8.33) - open/close 7338 (time 1780333414.70 total 597.47 last 14.52) - open/close 7496 (time 1780333424.71 total 607.48 last 15.79) - open/close 7617 (time 1780333435.57 total 618.34 last 11.15) - open/close 7761 (time 1780333446.32 total 629.09 last 13.39) - open/close 7895 (time 1780333456.74 total 639.52 last 12.86) - open/close 8099 (time 1780333467.33 total 650.10 last 19.27) - open/close 8186 (time 1780333480.06 total 662.83 last 6.84) - open/close 8326 (time 1780333490.60 total 673.37 last 13.28) - open/close 8504 (time 1780333501.51 total 684.28 last 16.32) - open/close 8701 (time 1780333511.51 total 694.28 last 19.69) - open/close 8818 (time 1780333521.54 total 704.31 last 11.67) - open/close 8941 (time 1780333531.59 total 714.37 last 12.23) - open/close 9083 (time 1780333541.61 total 724.38 last 14.18) - open/close 9249 (time 1780333551.73 total 734.50 last 16.40) - open/close 9412 (time 1780333564.63 total 747.40 last 12.64) - open/close 9561 (time 1780333574.77 total 757.54 last 14.69) - open/close 9797 (time 1780333584.77 total 767.55 last 23.59) - open/close 9996 (time 1780333594.95 total 777.72 last 19.56) - open/close 10100 (time 1780333605.11 total 787.88 last 10.23) - open/close 10242 (time 1780333615.17 total 797.94 last 14.11) - open/close 10476 (time 1780333625.58 total 808.35 last 22.48) - open/close 10689 (time 1780333635.76 total 818.53 last 20.93) - open/close 10792 (time 1780333646.65 total 829.42 last 9.46) - open/close 10895 (time 1780333657.08 total 839.85 last 9.87) - open/close 11127 (time 1780333667.09 total 849.86 last 23.18) - open/close 11340 (time 1780333679.36 total 862.13 last 17.36) - open/close 11551 (time 1780333689.64 total 872.41 last 20.52) - open/close 11674 (time 1780333700.54 total 883.31 last 11.28) - open/close 11819 (time 1780333711.21 total 893.99 last 13.58) - open/close 11900 (time 1780333721.73 total 904.50 last 7.70) - open/close 12054 (time 1780333731.73 total 914.51 last 15.39) - open/close 12104 (time 1780333742.15 total 924.92 last 4.80) - open/close 12232 (time 1780333753.22 total 936.00 last 11.56) - open/close 12369 (time 1780333763.83 total 946.60 last 12.91) - open/close 12525 (time 1780333774.86 total 957.63 last 14.15) - open/close 12726 (time 1780333785.51 total 968.28 last 18.86) - open/close 12804 (time 1780333796.41 total 979.18 last 7.16) - open/close 13040 (time 1780333806.42 total 989.19 last 23.58) - open/close 13178 (time 1780333816.84 total 999.62 last 13.24) - open/close 13408 (time 1780333827.76 total 1010.53 last 21.06) - open/close 13642 (time 1780333837.91 total 1020.68 last 23.06) - open/close 13806 (time 1780333847.91 total 1030.68 last 16.40) - open/close 13947 (time 1780333857.96 total 1040.73 last 14.04) - open/close 14064 (time 1780333868.30 total 1051.07 last 11.31) - open/close 14308 (time 1780333879.09 total 1061.87 last 22.60) - open/close 14491 (time 1780333889.20 total 1071.97 last 18.11) - open/close 14618 (time 1780333899.42 total 1082.20 last 12.42) - open/close 14772 (time 1780333909.43 total 1092.20 last 15.39) - open/close 14984 (time 1780333919.43 total 1102.20 last 21.20) - open/close 15607 (time 1780333929.45 total 1112.22 last 62.18) - open/close 16300 (time 1780333939.46 total 1122.23 last 69.25) - open/close 17029 (time 1780333949.47 total 1132.24 last 72.83) - open/close 17764 (time 1780333959.47 total 1142.24 last 73.48) - open/close 18428 (time 1780333969.48 total 1152.25 last 66.36) - open/close 19095 (time 1780333979.48 total 1162.25 last 66.66) - open/close 19769 (time 1780333989.50 total 1172.27 last 67.25) - open/close 20000 (time 1780333993.30 total 1176.07 last 60.90) - open/close 20674 (time 1780334003.30 total 1186.07 last 67.39) - open/close 21317 (time 1780334013.31 total 1196.08 last 64.23) - open/close 21960 (time 1780334023.32 total 1206.09 last 64.24) - open/close 22515 (time 1780334033.34 total 1216.12 last 55.36) - open/close 23133 (time 1780334043.35 total 1226.13 last 61.74) - open/close 23695 (time 1780334053.35 total 1236.13 last 56.20) - open/close 24382 (time 1780334063.36 total 1246.13 last 68.69) - open/close 25024 (time 1780334073.36 total 1256.13 last 64.15) - open/close 25681 (time 1780334083.37 total 1266.14 last 65.66) - open/close 26275 (time 1780334093.38 total 1276.15 last 59.34) - open/close 26884 (time 1780334103.39 total 1286.17 last 60.81) - open/close 27538 (time 1780334113.40 total 1296.18 last 65.33) - open/close 28148 (time 1780334123.42 total 1306.19 last 60.92) - open/close 28733 (time 1780334133.42 total 1316.20 last 58.46) - open/close 29335 (time 1780334143.44 total 1326.21 last 60.13) - open/close 29831 (time 1780334153.44 total 1336.22 last 49.56) - open/close 30000 (time 1780334157.17 total 1339.94 last 45.39) - open/close 30545 (time 1780334167.17 total 1349.94 last 54.47) - open/close 31204 (time 1780334177.18 total 1359.95 last 65.86) - open/close 31791 (time 1780334187.18 total 1369.95 last 58.69) - open/close 32454 (time 1780334197.23 total 1380.01 last 65.94) - open/close 33095 (time 1780334207.24 total 1390.01 last 64.08) - open/close 33691 (time 1780334217.24 total 1400.01 last 59.58) - open/close 34292 (time 1780334227.25 total 1410.02 last 60.07) - open/close 34956 (time 1780334237.25 total 1420.02 last 66.35) - open/close 35632 (time 1780334247.26 total 1430.03 last 67.56) - open/close 36167 (time 1780334257.27 total 1440.04 last 53.46) - open/close 36797 (time 1780334267.28 total 1450.05 last 62.90) - open/close 37447 (time 1780334277.30 total 1460.07 last 64.89) - open/close 37967 (time 1780334287.31 total 1470.08 last 51.97) - open/close 38564 (time 1780334297.32 total 1480.09 last 59.64) - open/close 39205 (time 1780334307.32 total 1490.10 last 64.05) - open/close 39861 (time 1780334317.34 total 1500.11 last 65.52) - open/close 40000 (time 1780334319.84 total 1502.61 last 55.62) - open/close 40636 (time 1780334329.85 total 1512.62 last 63.52) - open/close 41235 (time 1780334339.86 total 1522.63 last 59.83) - open/close 41852 (time 1780334349.87 total 1532.64 last 61.67) - open/close 42448 (time 1780334359.87 total 1542.65 last 59.55) - open/close 43044 (time 1780334369.88 total 1552.65 last 59.59) - open/close 43603 (time 1780334379.88 total 1562.65 last 55.87) - open/close 44152 (time 1780334389.90 total 1572.67 last 54.82) - open/close 44767 (time 1780334399.92 total 1582.69 last 61.35) - open/close 45435 (time 1780334409.93 total 1592.70 last 66.74) - open/close 46072 (time 1780334419.96 total 1602.73 last 63.50) - open/close 46701 (time 1780334429.96 total 1612.74 last 62.87) - open/close 47337 (time 1780334439.98 total 1622.76 last 63.48) - open/close 47956 (time 1780334449.99 total 1632.76 last 61.86) - open/close 48567 (time 1780334460.00 total 1642.78 last 61.01) - open/close 49115 (time 1780334470.01 total 1652.78 last 54.79) - open/close 49668 (time 1780334480.01 total 1662.78 last 55.27) total: 50000 open/close in 1668.48 seconds: 29.97 ops/second - unlinked 0 (time 1780334491 ; total 0 ; last 0) - unlinked 10000 (time 1780334660 ; total 169 ; last 169) - unlinked 20000 (time 1780334834 ; total 343 ; last 174) - unlinked 30000 (time 1780335001 ; total 510 ; last 167) - unlinked 40000 (time 1780335176 ; total 685 ; last 175) total: 50000 unlinks in 846 seconds: 59.101654 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg132-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg132-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg132-server stop mds service on oleg132-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg132-server unloading modules via unload_modules_local on: 'oleg132-server' oleg132-server: oleg132-server.virtnet: executing unload_modules_local oleg132-server: modules unloaded. pdsh@oleg132-client: oleg132-client: ssh exited with exit code 2 pdsh@oleg132-client: oleg132-server: ssh exited with exit code 2 pdsh@oleg132-client: oleg132-client: ssh exited with exit code 2 pdsh@oleg132-client: oleg132-server: ssh exited with exit code 2