== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:41:36 (1781275296) start mds service on oleg237-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions oleg237-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg237-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg237-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg237-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-OST0000 oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 125 (time 1781275379.20 total 10.02 last 12.47) - open/close 295 (time 1781275389.20 total 20.03 last 16.99) - open/close 470 (time 1781275399.23 total 30.05 last 17.45) - open/close 727 (time 1781275409.24 total 40.06 last 25.69) - open/close 1007 (time 1781275419.24 total 50.06 last 28.00) - open/close 1222 (time 1781275429.87 total 60.69 last 20.22) - open/close 1385 (time 1781275440.66 total 71.48 last 15.11) - open/close 1711 (time 1781275451.55 total 82.37 last 29.93) - open/close 1968 (time 1781275462.36 total 93.18 last 23.78) - open/close 2217 (time 1781275472.37 total 103.19 last 24.87) - open/close 2398 (time 1781275482.38 total 113.20 last 18.09) - open/close 2606 (time 1781275492.57 total 123.39 last 20.40) - open/close 2787 (time 1781275502.58 total 133.40 last 18.08) - open/close 3060 (time 1781275512.71 total 143.53 last 26.96) - open/close 3226 (time 1781275522.72 total 153.54 last 16.58) - open/close 3422 (time 1781275535.71 total 166.53 last 15.09) - open/close 3464 (time 1781275546.07 total 176.89 last 4.05) - open/close 3691 (time 1781275556.36 total 187.18 last 22.05) - open/close 3949 (time 1781275567.38 total 198.20 last 23.41) - open/close 4259 (time 1781275578.10 total 208.92 last 28.93) - open/close 4458 (time 1781275588.12 total 218.94 last 19.85) - open/close 4716 (time 1781275598.13 total 228.95 last 25.79) - open/close 4905 (time 1781275608.76 total 239.58 last 17.78) - open/close 5100 (time 1781275618.93 total 249.75 last 19.18) - open/close 5245 (time 1781275629.28 total 260.11 last 14.00) - open/close 5433 (time 1781275639.29 total 270.11 last 18.79) - open/close 5661 (time 1781275649.29 total 280.11 last 22.79) - open/close 5863 (time 1781275660.33 total 291.15 last 18.30) - open/close 6093 (time 1781275670.35 total 301.17 last 22.96) - open/close 6311 (time 1781275680.74 total 311.56 last 20.97) - open/close 6492 (time 1781275691.60 total 322.42 last 16.66) - open/close 6724 (time 1781275702.49 total 333.31 last 21.31) - open/close 6884 (time 1781275714.48 total 345.30 last 13.35) - open/close 7025 (time 1781275724.48 total 355.30 last 14.09) - open/close 7269 (time 1781275734.74 total 365.56 last 23.79) - open/close 7521 (time 1781275744.74 total 375.56 last 25.20) - open/close 7752 (time 1781275754.77 total 385.59 last 23.04) - open/close 7951 (time 1781275765.02 total 395.85 last 19.40) - open/close 8197 (time 1781275775.79 total 406.61 last 22.86) - open/close 8397 (time 1781275786.49 total 417.31 last 18.68) - open/close 8628 (time 1781275796.52 total 427.34 last 23.05) - open/close 8813 (time 1781275806.52 total 437.34 last 18.49) - open/close 9056 (time 1781275816.54 total 447.36 last 24.26) - open/close 9177 (time 1781275826.57 total 457.39 last 12.06) - open/close 9316 (time 1781275839.24 total 470.06 last 10.97) - open/close 9518 (time 1781275850.02 total 480.84 last 18.74) - open/close 9701 (time 1781275861.87 total 492.69 last 15.44) - open/close 9947 (time 1781275872.08 total 502.90 last 24.10) - open/close 10000 (time 1781275874.15 total 504.98 last 25.57) - open/close 10241 (time 1781275885.04 total 515.86 last 22.14) - open/close 10480 (time 1781275895.78 total 526.60 last 22.25) - open/close 10729 (time 1781275906.42 total 537.24 last 23.41) - open/close 10855 (time 1781275916.46 total 547.28 last 12.55) - open/close 11097 (time 1781275927.31 total 558.13 last 22.29) - open/close 11279 (time 1781275937.44 total 568.26 last 17.96) - open/close 11414 (time 1781275947.76 total 578.59 last 13.08) - open/close 11555 (time 1781275958.17 total 588.99 last 13.55) - open/close 11733 (time 1781275968.35 total 599.18 last 17.48) - open/close 11974 (time 1781275978.36 total 609.18 last 24.08) - open/close 12187 (time 1781275988.76 total 619.58 last 20.49) - open/close 12349 (time 1781275998.77 total 629.59 last 16.18) - open/close 12545 (time 1781276008.78 total 639.60 last 19.59) - open/close 12783 (time 1781276018.78 total 649.60 last 23.80) - open/close 13021 (time 1781276029.28 total 660.10 last 22.66) - open/close 13222 (time 1781276039.59 total 670.41 last 19.50) - open/close 13435 (time 1781276049.68 total 680.50 last 21.10) - open/close 13693 (time 1781276060.62 total 691.44 last 23.58) - open/close 13951 (time 1781276071.65 total 702.47 last 23.39) - open/close 14110 (time 1781276082.27 total 713.09 last 14.97) - open/close 14312 (time 1781276092.53 total 723.35 last 19.70) - open/close 14591 (time 1781276103.09 total 733.91 last 26.41) - open/close 14815 (time 1781276113.13 total 743.95 last 22.32) - open/close 15068 (time 1781276123.48 total 754.31 last 24.43) - open/close 15322 (time 1781276133.80 total 764.63 last 24.61) - open/close 15576 (time 1781276143.82 total 774.65 last 25.35) - open/close 15734 (time 1781276153.83 total 784.65 last 15.79) - open/close 15936 (time 1781276163.88 total 794.70 last 20.10) - open/close 16094 (time 1781276173.90 total 804.73 last 15.76) - open/close 16269 (time 1781276183.91 total 814.74 last 17.48) - open/close 16487 (time 1781276194.58 total 825.40 last 20.44) - open/close 16727 (time 1781276204.59 total 835.41 last 23.97) - open/close 16903 (time 1781276215.00 total 845.82 last 16.91) - open/close 17025 (time 1781276225.38 total 856.20 last 11.76) - open/close 17220 (time 1781276237.66 total 868.48 last 15.87) - open/close 17466 (time 1781276247.67 total 878.50 last 24.57) - open/close 17645 (time 1781276260.02 total 890.84 last 14.50) - open/close 17745 (time 1781276270.60 total 901.43 last 9.44) - open/close 17944 (time 1781276281.25 total 912.07 last 18.69) - open/close 18187 (time 1781276291.25 total 922.07 last 24.30) - open/close 18317 (time 1781276301.34 total 932.16 last 12.89) - open/close 18499 (time 1781276312.03 total 942.85 last 17.02) - open/close 18699 (time 1781276322.04 total 952.86 last 19.98) - open/close 18946 (time 1781276332.95 total 963.77 last 22.64) - open/close 19127 (time 1781276342.96 total 973.78 last 18.08) - open/close 19804 (time 1781276352.96 total 983.79 last 67.66) - open/close 20000 (time 1781276355.98 total 986.80 last 64.93) - open/close 20681 (time 1781276365.99 total 996.81 last 68.08) - open/close 21395 (time 1781276376.00 total 1006.82 last 71.30) - open/close 22220 (time 1781276386.01 total 1016.83 last 82.46) - open/close 23098 (time 1781276396.01 total 1026.84 last 87.73) - open/close 23895 (time 1781276406.02 total 1036.84 last 79.68) - open/close 24665 (time 1781276416.04 total 1046.86 last 76.83) - open/close 25503 (time 1781276426.04 total 1056.86 last 83.80) - open/close 26370 (time 1781276436.08 total 1066.90 last 86.38) - open/close 27202 (time 1781276446.08 total 1076.90 last 83.17) - open/close 28010 (time 1781276456.09 total 1086.91 last 80.69) - open/close 28724 (time 1781276466.09 total 1096.91 last 71.40) - open/close 29489 (time 1781276476.11 total 1106.93 last 76.36) - open/close 30000 (time 1781276482.22 total 1113.04 last 83.70) - open/close 30770 (time 1781276492.22 total 1123.05 last 76.94) - open/close 31482 (time 1781276502.24 total 1133.06 last 71.09) - open/close 32284 (time 1781276512.24 total 1143.07 last 80.16) - open/close 33121 (time 1781276522.25 total 1153.07 last 83.64) - open/close 33910 (time 1781276532.27 total 1163.09 last 78.76) - open/close 34755 (time 1781276542.27 total 1173.09 last 84.47) - open/close 35484 (time 1781276552.28 total 1183.10 last 72.87) - open/close 36272 (time 1781276562.29 total 1193.11 last 78.72) - open/close 37081 (time 1781276572.30 total 1203.12 last 80.83) - open/close 37848 (time 1781276582.31 total 1213.13 last 76.62) - open/close 38563 (time 1781276592.31 total 1223.13 last 71.46) - open/close 39335 (time 1781276602.31 total 1233.13 last 77.19) - open/close 39914 (time 1781276612.32 total 1243.14 last 57.84) - open/close 40664 (time 1781276622.33 total 1253.15 last 74.94) - open/close 41526 (time 1781276632.34 total 1263.16 last 86.17) - open/close 42279 (time 1781276642.34 total 1273.17 last 75.23) - open/close 43012 (time 1781276652.36 total 1283.18 last 73.21) - open/close 43733 (time 1781276662.36 total 1293.18 last 72.05) - open/close 44351 (time 1781276672.37 total 1303.19 last 61.75) - open/close 44913 (time 1781276682.37 total 1313.20 last 56.18) - open/close 45526 (time 1781276692.38 total 1323.20 last 61.28) - open/close 46540 (time 1781276702.38 total 1333.20 last 101.37) - open/close 47292 (time 1781276712.38 total 1343.20 last 75.19) - open/close 48055 (time 1781276722.39 total 1353.21 last 76.22) - open/close 48654 (time 1781276732.40 total 1363.22 last 59.88) - open/close 49487 (time 1781276742.40 total 1373.23 last 83.23) total: 50000 open/close in 1379.70 seconds: 36.24 ops/second - unlinked 0 (time 1781276753 ; total 0 ; last 0) - unlinked 10000 (time 1781276897 ; total 144 ; last 144) - unlinked 20000 (time 1781277022 ; total 269 ; last 125) - unlinked 30000 (time 1781277152 ; total 399 ; last 130) - unlinked 40000 (time 1781277302 ; total 549 ; last 150) total: 50000 unlinks in 698 seconds: 71.633240 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg237-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg237-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server stop mds service on oleg237-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg237-server unloading modules via unload_modules_local on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing unload_modules_local oleg237-server: modules unloaded. pdsh@oleg237-client: oleg237-client: ssh exited with exit code 2 pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2 pdsh@oleg237-client: oleg237-client: ssh exited with exit code 2 pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2