== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 11:25:06 (1787844306) start mds service on oleg126-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg126-server' oleg126-server: oleg126-server.virtnet: executing load_modules_local oleg126-server: Loading modules from /home/green/git/lustre-release/lustre oleg126-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg126-server: detected 4 online CPUs by sysfs oleg126-server: MODOPTS_LIBCFS= oleg126-server: Force libcfs to create 2 CPU partitions oleg126-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg126-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg126-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg126-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Started lustre-OST0000 oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg126-client.virtnet: -o user_xattr,flock 192.168.201.126@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 53 (time 1787844393.46 total 10.31 last 5.14) - open/close 211 (time 1787844404.46 total 21.32 last 14.36) - open/close 363 (time 1787844414.70 total 31.55 last 14.85) - open/close 603 (time 1787844424.70 total 41.56 last 23.99) - open/close 859 (time 1787844435.42 total 52.28 last 23.88) - open/close 1128 (time 1787844445.43 total 62.29 last 26.86) - open/close 1343 (time 1787844455.44 total 72.30 last 21.48) - open/close 1578 (time 1787844465.44 total 82.30 last 23.49) - open/close 1683 (time 1787844475.82 total 92.68 last 10.12) - open/close 1860 (time 1787844486.76 total 103.62 last 16.18) - open/close 1984 (time 1787844496.76 total 113.62 last 12.40) - open/close 2175 (time 1787844507.10 total 123.96 last 18.47) - open/close 2392 (time 1787844517.10 total 133.96 last 21.70) - open/close 2498 (time 1787844527.26 total 144.12 last 10.43) - open/close 2712 (time 1787844537.27 total 154.13 last 21.38) - open/close 2941 (time 1787844548.00 total 164.86 last 21.34) - open/close 3185 (time 1787844558.13 total 174.99 last 24.09) - open/close 3443 (time 1787844568.15 total 185.01 last 25.76) - open/close 3631 (time 1787844579.12 total 195.98 last 17.13) - open/close 3824 (time 1787844589.13 total 205.99 last 19.27) - open/close 3976 (time 1787844599.51 total 216.36 last 14.66) - open/close 4206 (time 1787844609.51 total 226.37 last 22.99) - open/close 4397 (time 1787844619.52 total 236.38 last 19.07) - open/close 4555 (time 1787844629.93 total 246.79 last 15.18) - open/close 4737 (time 1787844639.95 total 256.81 last 18.17) - open/close 4861 (time 1787844649.99 total 266.85 last 12.35) - open/close 5004 (time 1787844660.89 total 277.75 last 13.12) - open/close 5146 (time 1787844671.32 total 288.18 last 13.60) - open/close 5222 (time 1787844683.37 total 300.23 last 6.31) - open/close 5388 (time 1787844693.39 total 310.25 last 16.57) - open/close 5570 (time 1787844703.40 total 320.26 last 18.18) - open/close 5651 (time 1787844713.42 total 330.28 last 8.09) - open/close 5902 (time 1787844723.42 total 340.28 last 25.09) - open/close 6097 (time 1787844733.42 total 350.28 last 19.49) - open/close 6239 (time 1787844744.23 total 361.09 last 13.14) - open/close 6437 (time 1787844754.36 total 371.22 last 19.55) - open/close 6672 (time 1787844764.38 total 381.24 last 23.46) - open/close 6840 (time 1787844774.83 total 391.69 last 16.08) - open/close 7073 (time 1787844785.53 total 402.39 last 21.78) - open/close 7218 (time 1787844796.07 total 412.93 last 13.76) - open/close 7360 (time 1787844807.33 total 424.19 last 12.61) - open/close 7419 (time 1787844818.27 total 435.13 last 5.39) - open/close 7568 (time 1787844828.27 total 445.13 last 14.89) - open/close 7632 (time 1787844838.50 total 455.36 last 6.26) - open/close 7690 (time 1787844848.72 total 465.58 last 5.68) - open/close 7773 (time 1787844859.39 total 476.25 last 7.77) - open/close 7873 (time 1787844870.38 total 487.24 last 9.10) - open/close 7917 (time 1787844880.70 total 497.56 last 4.27) - open/close 7975 (time 1787844891.58 total 508.44 last 5.33) - open/close 8174 (time 1787844901.59 total 518.45 last 19.88) - open/close 8383 (time 1787844911.59 total 528.45 last 20.90) - open/close 8650 (time 1787844922.09 total 538.95 last 25.43) - open/close 8755 (time 1787844932.83 total 549.69 last 9.77) - open/close 8912 (time 1787844943.79 total 560.65 last 14.33) - open/close 9003 (time 1787844954.35 total 571.21 last 8.61) - open/close 9118 (time 1787844965.30 total 582.16 last 10.51) - open/close 9174 (time 1787844975.31 total 592.17 last 5.59) - open/close 9349 (time 1787844985.34 total 602.20 last 17.45) - open/close 9518 (time 1787844996.26 total 613.12 last 15.47) - open/close 9588 (time 1787845006.27 total 623.13 last 6.99) - open/close 9774 (time 1787845019.01 total 635.86 last 14.61) - open/close 9966 (time 1787845029.29 total 646.15 last 18.66) - open/close 10089 (time 1787845039.54 total 656.40 last 12.00) - open/close 10181 (time 1787845050.40 total 667.25 last 8.48) - open/close 10280 (time 1787845060.41 total 677.27 last 9.88) - open/close 10419 (time 1787845071.22 total 688.08 last 12.86) - open/close 10615 (time 1787845081.39 total 698.25 last 19.26) - open/close 10717 (time 1787845092.33 total 709.19 last 9.33) - open/close 10852 (time 1787845102.33 total 719.19 last 13.50) - open/close 11027 (time 1787845112.33 total 729.19 last 17.50) - open/close 11182 (time 1787845122.47 total 739.33 last 15.29) - open/close 11322 (time 1787845132.79 total 749.65 last 13.57) - open/close 11445 (time 1787845142.98 total 759.84 last 12.07) - open/close 11533 (time 1787845153.01 total 769.87 last 8.77) - open/close 11666 (time 1787845163.20 total 780.06 last 13.05) - open/close 11909 (time 1787845173.23 total 790.09 last 24.24) - open/close 12171 (time 1787845183.25 total 800.11 last 26.14) - open/close 12362 (time 1787845193.31 total 810.17 last 18.99) - open/close 12594 (time 1787845203.37 total 820.23 last 23.07) - open/close 12678 (time 1787845215.01 total 831.87 last 7.22) - open/close 12857 (time 1787845225.66 total 842.52 last 16.80) - open/close 13044 (time 1787845236.64 total 853.50 last 17.04) - open/close 13225 (time 1787845246.65 total 863.51 last 18.08) - open/close 13362 (time 1787845257.65 total 874.51 last 12.46) - open/close 13623 (time 1787845267.87 total 884.73 last 25.53) - open/close 13775 (time 1787845278.43 total 895.29 last 14.39) - open/close 13880 (time 1787845289.27 total 906.12 last 9.69) - open/close 13969 (time 1787845299.39 total 916.25 last 8.79) - open/close 14136 (time 1787845309.40 total 926.26 last 16.68) - open/close 14214 (time 1787845320.43 total 937.29 last 7.07) - open/close 14513 (time 1787845330.44 total 947.30 last 29.88) - open/close 14702 (time 1787845340.47 total 957.33 last 18.85) - open/close 14908 (time 1787845350.47 total 967.33 last 20.59) - open/close 15163 (time 1787845360.96 total 977.82 last 24.32) - open/close 15368 (time 1787845370.97 total 987.83 last 20.48) - open/close 15523 (time 1787845381.33 total 998.19 last 14.96) - open/close 15722 (time 1787845391.41 total 1008.27 last 19.74) - open/close 15931 (time 1787845401.42 total 1018.28 last 20.87) - open/close 16057 (time 1787845411.43 total 1028.29 last 12.59) - open/close 16196 (time 1787845424.45 total 1041.31 last 10.68) - open/close 16364 (time 1787845435.06 total 1051.92 last 15.83) - open/close 16772 (time 1787845445.09 total 1061.95 last 40.70) - open/close 17423 (time 1787845455.10 total 1071.96 last 65.02) - open/close 18111 (time 1787845465.10 total 1081.96 last 68.79) - open/close 18838 (time 1787845475.10 total 1091.96 last 72.67) - open/close 19433 (time 1787845485.11 total 1101.97 last 59.46) - open/close 20000 (time 1787845492.43 total 1109.29 last 77.50) - open/close 20688 (time 1787845502.44 total 1119.30 last 68.72) - open/close 21553 (time 1787845512.44 total 1129.30 last 86.45) - open/close 22414 (time 1787845522.45 total 1139.31 last 86.06) - open/close 23202 (time 1787845532.47 total 1149.32 last 78.67) - open/close 24137 (time 1787845542.47 total 1159.33 last 93.46) - open/close 25013 (time 1787845552.47 total 1169.33 last 87.58) - open/close 25822 (time 1787845562.47 total 1179.33 last 80.88) - open/close 26507 (time 1787845572.48 total 1189.34 last 68.46) - open/close 27313 (time 1787845582.48 total 1199.34 last 80.59) - open/close 27925 (time 1787845592.49 total 1209.35 last 61.18) - open/close 28699 (time 1787845602.51 total 1219.37 last 77.23) - open/close 29518 (time 1787845612.51 total 1229.37 last 81.86) - open/close 30000 (time 1787845619.43 total 1236.29 last 69.69) - open/close 30735 (time 1787845629.43 total 1246.29 last 73.48) - open/close 31579 (time 1787845639.45 total 1256.31 last 84.25) - open/close 32361 (time 1787845649.46 total 1266.32 last 78.10) - open/close 33196 (time 1787845659.47 total 1276.33 last 83.46) - open/close 34113 (time 1787845669.47 total 1286.33 last 91.64) - open/close 35010 (time 1787845679.48 total 1296.34 last 89.65) - open/close 35811 (time 1787845689.49 total 1306.35 last 79.99) - open/close 36659 (time 1787845699.50 total 1316.35 last 84.78) - open/close 37257 (time 1787845709.51 total 1326.37 last 59.71) - open/close 37854 (time 1787845719.52 total 1336.38 last 59.61) - open/close 38647 (time 1787845729.53 total 1346.38 last 79.29) - open/close 39494 (time 1787845739.53 total 1356.39 last 84.67) - open/close 40000 (time 1787845746.55 total 1363.41 last 72.10) - open/close 40730 (time 1787845756.56 total 1373.42 last 72.93) - open/close 41352 (time 1787845766.56 total 1383.42 last 62.17) - open/close 41978 (time 1787845776.57 total 1393.43 last 62.54) - open/close 42870 (time 1787845786.58 total 1403.44 last 89.12) - open/close 43518 (time 1787845796.61 total 1413.47 last 64.59) - open/close 44285 (time 1787845806.62 total 1423.48 last 76.64) - open/close 45230 (time 1787845816.62 total 1433.48 last 94.46) - open/close 46091 (time 1787845826.63 total 1443.49 last 86.05) - open/close 46914 (time 1787845836.63 total 1453.49 last 82.27) - open/close 47700 (time 1787845846.65 total 1463.51 last 78.48) - open/close 48468 (time 1787845856.65 total 1473.51 last 76.77) - open/close 49323 (time 1787845866.66 total 1483.52 last 85.40) total: 50000 open/close in 1491.14 seconds: 33.53 ops/second - unlinked 0 (time 1787845878 ; total 0 ; last 0) - unlinked 10000 (time 1787846007 ; total 129 ; last 129) - unlinked 20000 (time 1787846135 ; total 257 ; last 128) - unlinked 30000 (time 1787846266 ; total 388 ; last 131) - unlinked 40000 (time 1787846403 ; total 525 ; last 137) total: 50000 unlinks in 655 seconds: 76.335876 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg126-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg126-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg126-server stop mds service on oleg126-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg126-server unloading modules via unload_modules_local on: 'oleg126-server' oleg126-server: oleg126-server.virtnet: executing unload_modules_local oleg126-server: modules unloaded. pdsh@oleg126-client: oleg126-client: ssh exited with exit code 2 pdsh@oleg126-client: oleg126-server: ssh exited with exit code 2 pdsh@oleg126-client: oleg126-client: ssh exited with exit code 2 pdsh@oleg126-client: oleg126-server: ssh exited with exit code 2