== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:37:57 (1787380677) start mds service on oleg111-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions oleg111-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg111-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg111-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg111-server: oleg111-server.virtnet: executing set_default_debug -1 all pdsh@oleg111-client: oleg111-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg111-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg111-server: oleg111-server.virtnet: executing set_default_debug -1 all pdsh@oleg111-client: oleg111-server: ssh exited with exit code 1 Started lustre-OST0000 oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 222 (time 1787380775.27 total 10.18 last 21.80) - open/close 391 (time 1787380785.28 total 20.20 last 16.87) - open/close 559 (time 1787380795.72 total 30.63 last 16.10) - open/close 818 (time 1787380805.83 total 40.75 last 25.61) - open/close 1052 (time 1787380816.85 total 51.76 last 21.24) - open/close 1213 (time 1787380827.28 total 62.20 last 15.43) - open/close 1391 (time 1787380837.81 total 72.73 last 16.91) - open/close 1572 (time 1787380847.81 total 82.73 last 18.10) - open/close 1707 (time 1787380858.77 total 93.69 last 12.32) - open/close 1857 (time 1787380868.78 total 103.70 last 14.98) - open/close 2081 (time 1787380878.81 total 113.72 last 22.35) - open/close 2229 (time 1787380888.98 total 123.89 last 14.55) - open/close 2506 (time 1787380899.09 total 134.00 last 27.39) - open/close 2751 (time 1787380909.12 total 144.03 last 24.43) - open/close 2985 (time 1787380919.12 total 154.04 last 23.39) - open/close 3150 (time 1787380929.88 total 164.80 last 15.33) - open/close 3275 (time 1787380939.91 total 174.82 last 12.47) - open/close 3463 (time 1787380949.92 total 184.84 last 18.77) - open/close 3541 (time 1787380959.93 total 194.84 last 7.80) - open/close 3716 (time 1787380969.96 total 204.87 last 17.45) - open/close 3997 (time 1787380979.97 total 214.89 last 28.05) - open/close 4086 (time 1787380998.47 total 233.39 last 4.81) - open/close 4245 (time 1787381008.48 total 243.40 last 15.88) - open/close 4385 (time 1787381018.66 total 253.57 last 13.76) - open/close 4501 (time 1787381028.75 total 263.67 last 11.49) - open/close 4673 (time 1787381038.76 total 273.67 last 17.19) - open/close 4957 (time 1787381049.55 total 284.47 last 26.31) - open/close 5197 (time 1787381060.31 total 295.23 last 22.30) - open/close 5442 (time 1787381071.18 total 306.10 last 22.54) - open/close 5658 (time 1787381081.18 total 316.10 last 21.60) - open/close 5835 (time 1787381091.97 total 326.89 last 16.40) - open/close 6046 (time 1787381102.51 total 337.43 last 20.02) - open/close 6149 (time 1787381112.67 total 347.59 last 10.14) - open/close 6271 (time 1787381122.86 total 357.78 last 11.97) - open/close 6389 (time 1787381132.89 total 367.80 last 11.77) - open/close 6564 (time 1787381143.68 total 378.60 last 16.21) - open/close 6751 (time 1787381154.59 total 389.51 last 17.14) - open/close 6963 (time 1787381165.51 total 400.42 last 19.42) - open/close 7193 (time 1787381175.56 total 410.47 last 22.89) - open/close 7409 (time 1787381186.12 total 421.03 last 20.45) - open/close 7557 (time 1787381196.36 total 431.27 last 14.45) - open/close 7655 (time 1787381207.05 total 441.97 last 9.17) - open/close 7890 (time 1787381217.07 total 451.99 last 23.45) - open/close 8074 (time 1787381227.10 total 462.01 last 18.36) - open/close 8200 (time 1787381237.12 total 472.04 last 12.57) - open/close 8357 (time 1787381247.72 total 482.63 last 14.82) - open/close 8555 (time 1787381257.73 total 492.65 last 19.77) - open/close 8817 (time 1787381267.76 total 502.68 last 26.11) - open/close 8983 (time 1787381278.42 total 513.33 last 15.58) - open/close 9255 (time 1787381288.43 total 523.34 last 27.17) - open/close 9415 (time 1787381298.51 total 533.43 last 15.87) - open/close 9625 (time 1787381309.00 total 543.92 last 20.02) - open/close 9709 (time 1787381319.50 total 554.41 last 8.00) - open/close 9955 (time 1787381330.50 total 565.42 last 22.36) - open/close 10000 (time 1787381335.90 total 570.82 last 8.34) - open/close 10057 (time 1787381346.70 total 581.61 last 5.28) - open/close 10175 (time 1787381356.70 total 591.61 last 11.80) - open/close 10297 (time 1787381368.09 total 603.01 last 10.70) - open/close 10458 (time 1787381378.11 total 613.02 last 16.08) - open/close 10620 (time 1787381388.82 total 623.74 last 15.11) - open/close 10778 (time 1787381398.83 total 633.74 last 15.80) - open/close 10893 (time 1787381409.29 total 644.21 last 10.99) - open/close 11009 (time 1787381419.59 total 654.51 last 11.26) - open/close 11247 (time 1787381429.60 total 664.52 last 23.78) - open/close 11414 (time 1787381439.64 total 674.55 last 16.64) - open/close 11562 (time 1787381450.17 total 685.09 last 14.05) - open/close 11780 (time 1787381461.16 total 696.07 last 19.85) - open/close 11959 (time 1787381471.40 total 706.31 last 17.48) - open/close 12210 (time 1787381481.41 total 716.33 last 25.06) - open/close 12272 (time 1787381491.41 total 726.33 last 6.20) - open/close 12335 (time 1787381501.56 total 736.47 last 6.21) - open/close 12542 (time 1787381511.59 total 746.51 last 20.63) - open/close 12702 (time 1787381521.95 total 756.87 last 15.44) - open/close 12786 (time 1787381531.97 total 766.89 last 8.38) - open/close 12921 (time 1787381543.01 total 777.93 last 12.23) - open/close 13134 (time 1787381553.50 total 788.41 last 20.32) - open/close 13387 (time 1787381563.50 total 798.42 last 25.29) - open/close 13512 (time 1787381573.52 total 808.43 last 12.48) - open/close 13589 (time 1787381583.54 total 818.46 last 7.68) - open/close 13779 (time 1787381593.54 total 828.46 last 19.00) - open/close 13966 (time 1787381603.56 total 838.47 last 18.67) - open/close 14151 (time 1787381614.62 total 849.54 last 16.72) - open/close 14409 (time 1787381624.64 total 859.56 last 25.75) - open/close 14654 (time 1787381634.67 total 869.58 last 24.44) - open/close 14914 (time 1787381644.67 total 879.58 last 26.00) - open/close 15144 (time 1787381655.39 total 890.31 last 21.44) - open/close 15391 (time 1787381665.40 total 900.31 last 24.68) - open/close 15581 (time 1787381675.79 total 910.71 last 18.28) - open/close 15802 (time 1787381685.82 total 920.73 last 22.05) - open/close 16007 (time 1787381695.82 total 930.74 last 20.48) - open/close 16256 (time 1787381706.69 total 941.61 last 22.91) - open/close 16450 (time 1787381718.42 total 953.34 last 16.54) - open/close 16684 (time 1787381729.24 total 964.15 last 21.64) - open/close 16955 (time 1787381739.24 total 974.15 last 27.10) - open/close 17209 (time 1787381749.26 total 984.17 last 25.35) - open/close 17466 (time 1787381759.70 total 994.61 last 24.61) - open/close 17672 (time 1787381773.74 total 1008.66 last 14.67) - open/close 17829 (time 1787381783.75 total 1018.67 last 15.68) - open/close 18097 (time 1787381793.76 total 1028.68 last 26.77) - open/close 18935 (time 1787381803.77 total 1038.69 last 83.74) - open/close 19738 (time 1787381813.77 total 1048.69 last 80.27) - open/close 20000 (time 1787381816.75 total 1051.66 last 88.12) - open/close 20795 (time 1787381826.76 total 1061.68 last 79.38) - open/close 21523 (time 1787381836.76 total 1071.68 last 72.79) - open/close 22293 (time 1787381846.80 total 1081.71 last 76.75) - open/close 22990 (time 1787381856.80 total 1091.71 last 69.70) - open/close 23744 (time 1787381866.80 total 1101.72 last 75.37) - open/close 24571 (time 1787381876.80 total 1111.72 last 82.70) - open/close 25450 (time 1787381886.80 total 1121.72 last 87.89) - open/close 26316 (time 1787381896.80 total 1131.72 last 86.57) - open/close 27197 (time 1787381906.81 total 1141.73 last 88.05) - open/close 28019 (time 1787381916.81 total 1151.73 last 82.17) - open/close 28655 (time 1787381926.83 total 1161.75 last 63.47) - open/close 29186 (time 1787381936.84 total 1171.76 last 53.07) - open/close 29890 (time 1787381946.84 total 1181.76 last 70.39) - open/close 30671 (time 1787381956.85 total 1191.76 last 78.07) - open/close 31396 (time 1787381966.86 total 1201.77 last 72.42) - open/close 32184 (time 1787381976.86 total 1211.78 last 78.77) - open/close 32984 (time 1787381986.86 total 1221.78 last 79.98) - open/close 33870 (time 1787381996.87 total 1231.79 last 88.53) - open/close 34700 (time 1787382006.87 total 1241.79 last 82.96) - open/close 35594 (time 1787382016.90 total 1251.81 last 89.20) - open/close 36470 (time 1787382026.90 total 1261.82 last 87.55) - open/close 37355 (time 1787382036.91 total 1271.82 last 88.48) - open/close 38159 (time 1787382046.91 total 1281.83 last 80.33) - open/close 39006 (time 1787382056.92 total 1291.84 last 84.64) - open/close 39919 (time 1787382066.92 total 1301.84 last 91.30) - open/close 40759 (time 1787382076.93 total 1311.84 last 83.97) - open/close 41623 (time 1787382086.93 total 1321.85 last 86.34) - open/close 42559 (time 1787382096.94 total 1331.86 last 93.49) - open/close 43454 (time 1787382106.95 total 1341.87 last 89.45) - open/close 44184 (time 1787382116.96 total 1351.87 last 72.94) - open/close 44968 (time 1787382126.96 total 1361.88 last 78.39) - open/close 45810 (time 1787382136.97 total 1371.89 last 84.10) - open/close 46684 (time 1787382146.97 total 1381.89 last 87.39) - open/close 47580 (time 1787382156.98 total 1391.90 last 89.50) - open/close 48374 (time 1787382166.99 total 1401.91 last 79.35) - open/close 49112 (time 1787382177.00 total 1411.92 last 73.73) - open/close 49998 (time 1787382187.01 total 1421.92 last 88.56) total: 50000 open/close in 1421.94 seconds: 35.16 ops/second - unlinked 0 (time 1787382191 ; total 0 ; last 0) - unlinked 10000 (time 1787382320 ; total 129 ; last 129) - unlinked 20000 (time 1787382443 ; total 252 ; last 123) - unlinked 30000 (time 1787382565 ; total 374 ; last 122) - unlinked 40000 (time 1787382701 ; total 510 ; last 136) total: 50000 unlinks in 645 seconds: 77.519379 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg111-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg111-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg111-server stop mds service on oleg111-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg111-server unloading modules via unload_modules_local on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing unload_modules_local oleg111-server: modules unloaded. pdsh@oleg111-client: oleg111-client: ssh exited with exit code 2 pdsh@oleg111-client: oleg111-server: ssh exited with exit code 2 pdsh@oleg111-client: oleg111-client: ssh exited with exit code 2 pdsh@oleg111-client: oleg111-server: ssh exited with exit code 2