== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:12:37 (1783138357) start mds service on oleg247-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg247-server' oleg247-server: oleg247-server.virtnet: executing load_modules_local oleg247-server: Loading modules from /home/green/git/lustre-release/lustre oleg247-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg247-server: detected 4 online CPUs by sysfs oleg247-server: MODOPTS_LIBCFS= oleg247-server: Force libcfs to create 2 CPU partitions oleg247-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg247-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg247-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg247-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-OST0000 oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg247-client.virtnet: -o user_xattr,flock 192.168.202.147@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 131 (time 1783138453.71 total 10.00 last 13.10) - open/close 231 (time 1783138464.53 total 20.83 last 9.23) - open/close 421 (time 1783138475.40 total 31.69 last 17.49) - open/close 505 (time 1783138486.08 total 42.38 last 7.86) - open/close 651 (time 1783138497.05 total 53.34 last 13.32) - open/close 808 (time 1783138509.76 total 66.05 last 12.35) - open/close 897 (time 1783138520.52 total 76.81 last 8.27) - open/close 1014 (time 1783138530.82 total 87.12 last 11.36) - open/close 1141 (time 1783138541.72 total 98.01 last 11.66) - open/close 1307 (time 1783138552.07 total 108.36 last 16.04) - open/close 1502 (time 1783138562.30 total 118.60 last 19.05) - open/close 1679 (time 1783138572.31 total 128.61 last 17.68) - open/close 1869 (time 1783138582.34 total 138.63 last 18.95) - open/close 2003 (time 1783138592.75 total 149.04 last 12.87) - open/close 2092 (time 1783138602.76 total 159.06 last 8.89) - open/close 2180 (time 1783138612.78 total 169.08 last 8.78) - open/close 2420 (time 1783138622.80 total 179.10 last 23.95) - open/close 2544 (time 1783138634.01 total 190.31 last 11.06) - open/close 2749 (time 1783138644.30 total 200.59 last 19.93) - open/close 2953 (time 1783138654.32 total 210.62 last 20.34) - open/close 3112 (time 1783138664.72 total 221.02 last 15.29) - open/close 3278 (time 1783138675.47 total 231.76 last 15.45) - open/close 3424 (time 1783138686.14 total 242.44 last 13.68) - open/close 3603 (time 1783138696.15 total 252.44 last 17.89) - open/close 3719 (time 1783138706.15 total 262.44 last 11.60) - open/close 3948 (time 1783138716.17 total 272.46 last 22.86) - open/close 4168 (time 1783138726.18 total 282.47 last 21.97) - open/close 4356 (time 1783138736.19 total 292.48 last 18.78) - open/close 4447 (time 1783138746.95 total 303.25 last 8.45) - open/close 4591 (time 1783138757.14 total 313.44 last 14.13) - open/close 4667 (time 1783138767.40 total 323.69 last 7.41) - open/close 4735 (time 1783138778.40 total 334.70 last 6.18) - open/close 4939 (time 1783138789.19 total 345.49 last 18.91) - open/close 5144 (time 1783138799.87 total 356.16 last 19.20) - open/close 5352 (time 1783138810.42 total 366.72 last 19.71) - open/close 5607 (time 1783138821.44 total 377.74 last 23.15) - open/close 5869 (time 1783138832.28 total 388.58 last 24.16) - open/close 6137 (time 1783138842.29 total 398.58 last 26.80) - open/close 6339 (time 1783138852.29 total 408.59 last 20.19) - open/close 6526 (time 1783138862.30 total 418.59 last 18.69) - open/close 6769 (time 1783138872.40 total 428.69 last 24.06) - open/close 6989 (time 1783138882.93 total 439.23 last 20.88) - open/close 7143 (time 1783138892.94 total 449.24 last 15.38) - open/close 7319 (time 1783138904.00 total 460.30 last 15.91) - open/close 7471 (time 1783138914.97 total 471.27 last 13.86) - open/close 7710 (time 1783138924.98 total 481.27 last 23.90) - open/close 7907 (time 1783138935.67 total 491.97 last 18.42) - open/close 8174 (time 1783138946.71 total 503.01 last 24.18) - open/close 8385 (time 1783138957.14 total 513.44 last 20.23) - open/close 8629 (time 1783138967.69 total 523.99 last 23.13) - open/close 8893 (time 1783138978.16 total 534.46 last 25.20) - open/close 9132 (time 1783138988.87 total 545.16 last 22.33) - open/close 9327 (time 1783138998.90 total 555.19 last 19.44) - open/close 9510 (time 1783139009.50 total 565.80 last 17.26) - open/close 9711 (time 1783139019.68 total 575.97 last 19.75) - open/close 9981 (time 1783139030.72 total 587.02 last 24.45) - open/close 10161 (time 1783139040.74 total 597.03 last 17.98) - open/close 10270 (time 1783139050.74 total 607.03 last 10.90) - open/close 10356 (time 1783139060.74 total 617.04 last 8.60) - open/close 10493 (time 1783139071.07 total 627.37 last 13.26) - open/close 10742 (time 1783139081.09 total 637.38 last 24.86) - open/close 10956 (time 1783139091.33 total 647.62 last 20.90) - open/close 11204 (time 1783139101.33 total 657.63 last 24.79) - open/close 11419 (time 1783139111.56 total 667.86 last 21.02) - open/close 11598 (time 1783139122.19 total 678.49 last 16.84) - open/close 11849 (time 1783139133.07 total 689.37 last 23.06) - open/close 11973 (time 1783139143.51 total 699.81 last 11.88) - open/close 12217 (time 1783139153.51 total 709.81 last 24.40) - open/close 12444 (time 1783139164.48 total 720.77 last 20.70) - open/close 12562 (time 1783139174.49 total 730.78 last 11.79) - open/close 12689 (time 1783139184.49 total 740.78 last 12.70) - open/close 12897 (time 1783139195.46 total 751.76 last 18.95) - open/close 13159 (time 1783139205.47 total 761.76 last 26.19) - open/close 13357 (time 1783139215.47 total 771.77 last 19.79) - open/close 13531 (time 1783139225.50 total 781.79 last 17.36) - open/close 13636 (time 1783139236.68 total 792.98 last 9.38) - open/close 13800 (time 1783139246.69 total 802.98 last 16.39) - open/close 13837 (time 1783139257.03 total 813.33 last 3.58) - open/close 13917 (time 1783139267.16 total 823.46 last 7.90) - open/close 13968 (time 1783139277.82 total 834.12 last 4.78) - open/close 14055 (time 1783139288.16 total 844.46 last 8.41) - open/close 14254 (time 1783139299.22 total 855.51 last 18.00) - open/close 14414 (time 1783139309.22 total 865.52 last 15.99) - open/close 14639 (time 1783139319.23 total 875.53 last 22.48) - open/close 14765 (time 1783139329.25 total 885.55 last 12.58) - open/close 14950 (time 1783139339.66 total 895.95 last 17.78) - open/close 15149 (time 1783139350.61 total 906.90 last 18.17) - open/close 15412 (time 1783139361.47 total 917.77 last 24.21) - open/close 15613 (time 1783139372.21 total 928.51 last 18.71) - open/close 15794 (time 1783139383.60 total 939.89 last 15.90) - open/close 15989 (time 1783139393.62 total 949.91 last 19.46) - open/close 16148 (time 1783139404.70 total 960.99 last 14.35) - open/close 16375 (time 1783139415.43 total 971.72 last 21.16) - open/close 16608 (time 1783139426.38 total 982.67 last 21.28) - open/close 16837 (time 1783139437.48 total 993.77 last 20.63) - open/close 17021 (time 1783139447.50 total 1003.79 last 18.36) - open/close 17218 (time 1783139457.51 total 1013.81 last 19.66) - open/close 17739 (time 1783139467.52 total 1023.81 last 52.10) - open/close 18407 (time 1783139477.52 total 1033.81 last 66.78) - open/close 19007 (time 1783139487.53 total 1043.83 last 59.92) - open/close 19556 (time 1783139497.56 total 1053.85 last 54.75) - open/close 20000 (time 1783139506.30 total 1062.60 last 50.79) - open/close 20608 (time 1783139516.31 total 1072.61 last 60.72) - open/close 21266 (time 1783139526.32 total 1082.62 last 65.75) - open/close 22006 (time 1783139536.33 total 1092.62 last 73.94) - open/close 22736 (time 1783139546.33 total 1102.63 last 72.99) - open/close 23389 (time 1783139556.34 total 1112.63 last 65.27) - open/close 24082 (time 1783139566.35 total 1122.64 last 69.22) - open/close 24717 (time 1783139576.36 total 1132.65 last 63.42) - open/close 25404 (time 1783139586.37 total 1142.66 last 68.63) - open/close 26033 (time 1783139596.45 total 1152.75 last 62.39) - open/close 26547 (time 1783139606.45 total 1162.75 last 51.40) - open/close 27253 (time 1783139616.46 total 1172.76 last 70.51) - open/close 27972 (time 1783139626.48 total 1182.78 last 71.77) - open/close 28600 (time 1783139636.49 total 1192.78 last 62.77) - open/close 29269 (time 1783139646.50 total 1202.79 last 66.82) - open/close 29918 (time 1783139656.50 total 1212.80 last 64.86) - open/close 30608 (time 1783139666.51 total 1222.81 last 68.94) - open/close 31362 (time 1783139676.52 total 1232.82 last 75.31) - open/close 32154 (time 1783139686.53 total 1242.83 last 79.13) - open/close 32807 (time 1783139696.54 total 1252.83 last 65.25) - open/close 33424 (time 1783139706.54 total 1262.84 last 61.69) - open/close 34103 (time 1783139716.56 total 1272.86 last 67.74) - open/close 34612 (time 1783139726.60 total 1282.90 last 50.70) - open/close 35041 (time 1783139736.61 total 1292.90 last 42.89) - open/close 35665 (time 1783139746.61 total 1302.90 last 62.40) - open/close 36299 (time 1783139756.62 total 1312.92 last 63.31) - open/close 36996 (time 1783139766.63 total 1322.92 last 69.66) - open/close 37806 (time 1783139776.64 total 1332.93 last 80.91) - open/close 38576 (time 1783139786.65 total 1342.95 last 76.90) - open/close 39326 (time 1783139796.66 total 1352.95 last 74.95) - open/close 40000 (time 1783139806.40 total 1362.70 last 69.19) - open/close 40713 (time 1783139816.41 total 1372.70 last 71.24) - open/close 41362 (time 1783139826.41 total 1382.71 last 64.87) - open/close 42008 (time 1783139836.42 total 1392.72 last 64.55) - open/close 42652 (time 1783139846.44 total 1402.73 last 64.29) - open/close 43207 (time 1783139856.45 total 1412.75 last 55.41) - open/close 43831 (time 1783139866.46 total 1422.76 last 62.36) - open/close 44400 (time 1783139876.47 total 1432.76 last 56.86) - open/close 45048 (time 1783139886.47 total 1442.77 last 64.78) - open/close 45655 (time 1783139896.49 total 1452.79 last 60.58) - open/close 46213 (time 1783139906.50 total 1462.79 last 55.76) - open/close 46768 (time 1783139916.52 total 1472.81 last 55.39) - open/close 47369 (time 1783139926.52 total 1482.82 last 60.05) - open/close 47991 (time 1783139936.53 total 1492.83 last 62.16) - open/close 48650 (time 1783139946.53 total 1502.83 last 65.90) - open/close 49242 (time 1783139956.53 total 1512.83 last 59.19) - open/close 49877 (time 1783139966.56 total 1522.86 last 63.31) total: 50000 open/close in 1525.27 seconds: 32.78 ops/second - unlinked 0 (time 1783139974 ; total 0 ; last 0) - unlinked 10000 (time 1783140141 ; total 167 ; last 167) - unlinked 20000 (time 1783140313 ; total 339 ; last 172) - unlinked 30000 (time 1783140469 ; total 495 ; last 156) - unlinked 40000 (time 1783140613 ; total 639 ; last 144) total: 50000 unlinks in 774 seconds: 64.599480 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg247-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg247-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg247-server stop mds service on oleg247-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg247-server unloading modules via unload_modules_local on: 'oleg247-server' oleg247-server: oleg247-server.virtnet: executing unload_modules_local oleg247-server: modules unloaded. pdsh@oleg247-client: oleg247-server: ssh exited with exit code 2 pdsh@oleg247-client: oleg247-client: ssh exited with exit code 2 pdsh@oleg247-client: oleg247-client: ssh exited with exit code 2 pdsh@oleg247-client: oleg247-server: ssh exited with exit code 2