== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:22:24 (1786170144) start mds service on oleg305-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg305-server' oleg305-server: oleg305-server.virtnet: executing load_modules_local oleg305-server: Loading modules from /home/green/git/lustre-release/lustre oleg305-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg305-server: detected 4 online CPUs by sysfs oleg305-server: MODOPTS_LIBCFS= oleg305-server: Force libcfs to create 2 CPU partitions oleg305-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg305-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg305-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg305-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-OST0000 oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 178 (time 1786170228.77 total 10.28 last 17.32) - open/close 437 (time 1786170238.78 total 20.28 last 25.89) - open/close 549 (time 1786170248.78 total 30.29 last 11.19) - open/close 705 (time 1786170258.96 total 40.46 last 15.33) - open/close 812 (time 1786170269.90 total 51.40 last 9.78) - open/close 987 (time 1786170280.31 total 61.81 last 16.81) - open/close 1217 (time 1786170290.33 total 71.84 last 22.94) - open/close 1456 (time 1786170300.71 total 82.22 last 23.02) - open/close 1658 (time 1786170310.72 total 92.22 last 20.19) - open/close 1805 (time 1786170321.29 total 102.79 last 13.91) - open/close 2015 (time 1786170331.77 total 113.28 last 20.03) - open/close 2234 (time 1786170341.79 total 123.30 last 21.86) - open/close 2475 (time 1786170352.46 total 133.97 last 22.58) - open/close 2689 (time 1786170362.47 total 143.98 last 21.38) - open/close 2841 (time 1786170372.47 total 153.98 last 15.20) - open/close 3080 (time 1786170383.44 total 164.94 last 21.80) - open/close 3270 (time 1786170394.24 total 175.75 last 17.58) - open/close 3498 (time 1786170404.25 total 185.76 last 22.78) - open/close 3681 (time 1786170414.78 total 196.28 last 17.39) - open/close 3929 (time 1786170425.68 total 207.18 last 22.76) - open/close 4129 (time 1786170435.70 total 217.20 last 19.96) - open/close 4291 (time 1786170445.98 total 227.48 last 15.76) - open/close 4459 (time 1786170456.00 total 237.50 last 16.77) - open/close 4636 (time 1786170466.66 total 248.17 last 16.59) - open/close 4845 (time 1786170477.85 total 259.36 last 18.68) - open/close 5004 (time 1786170487.87 total 269.37 last 15.88) - open/close 5226 (time 1786170498.47 total 279.97 last 20.94) - open/close 5506 (time 1786170508.94 total 290.45 last 26.73) - open/close 5686 (time 1786170519.17 total 300.68 last 17.60) - open/close 5944 (time 1786170530.17 total 311.67 last 23.46) - open/close 6157 (time 1786170540.62 total 322.13 last 20.38) - open/close 6354 (time 1786170551.03 total 332.54 last 18.92) - open/close 6590 (time 1786170561.04 total 342.54 last 23.59) - open/close 6857 (time 1786170571.05 total 352.56 last 26.65) - open/close 6973 (time 1786170581.38 total 362.88 last 11.24) - open/close 7157 (time 1786170593.64 total 375.14 last 15.01) - open/close 7330 (time 1786170606.50 total 388.01 last 13.44) - open/close 7493 (time 1786170616.52 total 398.03 last 16.28) - open/close 7692 (time 1786170626.53 total 408.03 last 19.89) - open/close 7898 (time 1786170637.60 total 419.10 last 18.61) - open/close 8110 (time 1786170647.60 total 429.10 last 21.20) - open/close 8308 (time 1786170657.98 total 439.49 last 19.07) - open/close 8552 (time 1786170668.92 total 450.43 last 22.30) - open/close 8783 (time 1786170678.93 total 460.43 last 23.09) - open/close 9008 (time 1786170688.95 total 470.46 last 22.44) - open/close 9213 (time 1786170698.97 total 480.48 last 20.47) - open/close 9430 (time 1786170709.01 total 490.52 last 21.61) - open/close 9601 (time 1786170719.13 total 500.64 last 16.90) - open/close 9722 (time 1786170729.15 total 510.66 last 12.07) - open/close 9915 (time 1786170740.15 total 521.65 last 17.56) - open/close 10000 (time 1786170745.61 total 527.11 last 15.57) - open/close 10247 (time 1786170756.14 total 537.65 last 23.44) - open/close 10437 (time 1786170766.23 total 547.74 last 18.83) - open/close 10601 (time 1786170776.24 total 557.75 last 16.39) - open/close 10802 (time 1786170786.25 total 567.76 last 20.07) - open/close 11012 (time 1786170796.55 total 578.05 last 20.40) - open/close 11216 (time 1786170806.57 total 588.07 last 20.36) - open/close 11387 (time 1786170816.57 total 598.08 last 17.10) - open/close 11518 (time 1786170826.58 total 608.08 last 13.09) - open/close 11713 (time 1786170836.91 total 618.42 last 18.87) - open/close 11822 (time 1786170847.38 total 628.89 last 10.41) - open/close 11980 (time 1786170857.42 total 638.93 last 15.73) - open/close 12210 (time 1786170867.43 total 648.94 last 22.98) - open/close 12394 (time 1786170877.95 total 659.45 last 17.50) - open/close 12531 (time 1786170888.29 total 669.80 last 13.24) - open/close 12725 (time 1786170898.73 total 680.23 last 18.59) - open/close 12879 (time 1786170909.78 total 691.29 last 13.93) - open/close 13088 (time 1786170920.14 total 701.64 last 20.18) - open/close 13292 (time 1786170931.13 total 712.64 last 18.56) - open/close 13533 (time 1786170941.14 total 722.64 last 24.09) - open/close 13725 (time 1786170951.14 total 732.65 last 19.19) - open/close 13936 (time 1786170961.92 total 743.43 last 19.57) - open/close 14079 (time 1786170972.61 total 754.11 last 13.38) - open/close 14258 (time 1786170983.09 total 764.60 last 17.08) - open/close 14464 (time 1786170993.14 total 774.65 last 20.49) - open/close 14652 (time 1786171003.15 total 784.66 last 18.78) - open/close 14783 (time 1786171013.31 total 794.81 last 12.90) - open/close 14926 (time 1786171023.58 total 805.09 last 13.92) - open/close 15142 (time 1786171033.59 total 815.09 last 21.59) - open/close 15267 (time 1786171044.49 total 826.00 last 11.46) - open/close 15467 (time 1786171055.14 total 836.65 last 18.78) - open/close 15677 (time 1786171065.15 total 846.66 last 20.98) - open/close 15905 (time 1786171075.55 total 857.06 last 21.93) - open/close 16069 (time 1786171086.03 total 867.53 last 15.65) - open/close 16287 (time 1786171096.08 total 877.59 last 21.68) - open/close 16462 (time 1786171106.10 total 887.61 last 17.46) - open/close 16594 (time 1786171116.13 total 897.64 last 13.17) - open/close 16863 (time 1786171126.18 total 907.69 last 26.76) - open/close 17052 (time 1786171136.25 total 917.75 last 18.78) - open/close 17238 (time 1786171146.28 total 927.78 last 18.54) - open/close 17451 (time 1786171156.31 total 937.82 last 21.23) - open/close 17661 (time 1786171167.29 total 948.80 last 19.13) - open/close 17792 (time 1786171178.27 total 959.78 last 11.93) - open/close 17915 (time 1786171188.29 total 969.80 last 12.28) - open/close 18105 (time 1786171199.23 total 980.73 last 17.37) - open/close 18285 (time 1786171209.62 total 991.12 last 17.33) - open/close 18388 (time 1786171220.41 total 1001.92 last 9.54) - open/close 18632 (time 1786171230.41 total 1011.92 last 24.39) - open/close 19147 (time 1786171240.42 total 1021.93 last 51.46) - open/close 19736 (time 1786171250.43 total 1031.94 last 58.86) - open/close 20000 (time 1786171254.23 total 1035.74 last 69.42) - open/close 20737 (time 1786171264.24 total 1045.74 last 73.66) - open/close 21495 (time 1786171274.25 total 1055.76 last 75.71) - open/close 22230 (time 1786171284.25 total 1065.76 last 73.47) - open/close 22919 (time 1786171294.26 total 1075.77 last 68.86) - open/close 23627 (time 1786171304.26 total 1085.77 last 70.79) - open/close 24318 (time 1786171314.28 total 1095.78 last 69.01) - open/close 24944 (time 1786171324.29 total 1105.80 last 62.52) - open/close 25597 (time 1786171334.30 total 1115.81 last 65.20) - open/close 26236 (time 1786171344.30 total 1125.81 last 63.90) - open/close 26804 (time 1786171354.31 total 1135.81 last 56.77) - open/close 27432 (time 1786171364.32 total 1145.82 last 62.75) - open/close 28101 (time 1786171374.32 total 1155.83 last 66.88) - open/close 28693 (time 1786171384.34 total 1165.84 last 59.11) - open/close 29335 (time 1786171394.34 total 1175.85 last 64.16) - open/close 29945 (time 1786171404.35 total 1185.86 last 60.94) - open/close 30567 (time 1786171414.36 total 1195.86 last 62.17) - open/close 31139 (time 1786171424.36 total 1205.86 last 57.19) - open/close 31785 (time 1786171434.44 total 1215.94 last 64.09) - open/close 32437 (time 1786171444.47 total 1225.98 last 64.98) - open/close 33124 (time 1786171454.47 total 1235.98 last 68.69) - open/close 33775 (time 1786171464.49 total 1245.99 last 65.03) - open/close 34436 (time 1786171474.52 total 1256.02 last 65.90) - open/close 35110 (time 1786171484.52 total 1266.03 last 67.34) - open/close 35724 (time 1786171494.53 total 1276.04 last 61.36) - open/close 36403 (time 1786171504.54 total 1286.05 last 67.83) - open/close 37097 (time 1786171514.55 total 1296.05 last 69.37) - open/close 37819 (time 1786171524.59 total 1306.10 last 71.87) - open/close 38508 (time 1786171534.61 total 1316.12 last 68.78) - open/close 39125 (time 1786171544.62 total 1326.13 last 61.63) - open/close 39808 (time 1786171554.62 total 1336.13 last 68.29) - open/close 40000 (time 1786171557.85 total 1339.36 last 59.40) - open/close 40722 (time 1786171567.88 total 1349.39 last 71.98) - open/close 41452 (time 1786171577.89 total 1359.39 last 72.98) - open/close 42133 (time 1786171587.89 total 1369.40 last 68.07) - open/close 42768 (time 1786171597.90 total 1379.41 last 63.45) - open/close 43391 (time 1786171607.90 total 1389.41 last 62.29) - open/close 44053 (time 1786171617.90 total 1399.41 last 66.18) - open/close 44724 (time 1786171627.92 total 1409.42 last 67.01) - open/close 45392 (time 1786171637.93 total 1419.44 last 66.71) - open/close 46096 (time 1786171647.94 total 1429.44 last 70.36) - open/close 46810 (time 1786171657.96 total 1439.46 last 71.25) - open/close 47476 (time 1786171667.96 total 1449.47 last 66.58) - open/close 48178 (time 1786171677.96 total 1459.47 last 70.18) - open/close 48827 (time 1786171687.97 total 1469.47 last 64.89) - open/close 49331 (time 1786171697.97 total 1479.47 last 50.40) total: 50000 open/close in 1489.19 seconds: 33.58 ops/second - unlinked 0 (time 1786171711 ; total 0 ; last 0) - unlinked 10000 (time 1786171874 ; total 163 ; last 163) - unlinked 20000 (time 1786172044 ; total 333 ; last 170) - unlinked 30000 (time 1786172208 ; total 497 ; last 164) - unlinked 40000 (time 1786172369 ; total 658 ; last 161) total: 50000 unlinks in 820 seconds: 60.975609 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg305-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg305-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg305-server stop mds service on oleg305-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg305-server unloading modules via unload_modules_local on: 'oleg305-server' oleg305-server: oleg305-server.virtnet: executing unload_modules_local oleg305-server: modules unloaded. pdsh@oleg305-client: oleg305-client: ssh exited with exit code 2 pdsh@oleg305-client: oleg305-server: ssh exited with exit code 2 pdsh@oleg305-client: oleg305-client: ssh exited with exit code 2 pdsh@oleg305-client: oleg305-server: ssh exited with exit code 2