== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 04:49:51 (1768556991) start mds service on oleg254-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions oleg254-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg254-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg254-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 265 (time 1768557061.73 total 10.56 last 25.11) - open/close 485 (time 1768557072.61 total 21.44 last 20.22) - open/close 692 (time 1768557082.61 total 31.44 last 20.70) - open/close 902 (time 1768557092.76 total 41.59 last 20.69) - open/close 1137 (time 1768557103.00 total 51.83 last 22.94) - open/close 1353 (time 1768557113.02 total 61.85 last 21.56) - open/close 1584 (time 1768557123.05 total 71.88 last 23.04) - open/close 1804 (time 1768557133.06 total 81.89 last 21.97) - open/close 1983 (time 1768557143.47 total 92.30 last 17.21) - open/close 2240 (time 1768557154.01 total 102.84 last 24.37) - open/close 2507 (time 1768557164.02 total 112.85 last 26.67) - open/close 2740 (time 1768557174.03 total 122.86 last 23.29) - open/close 2972 (time 1768557184.14 total 132.97 last 22.94) - open/close 3240 (time 1768557195.09 total 143.92 last 24.48) - open/close 3471 (time 1768557205.10 total 153.93 last 23.09) - open/close 3690 (time 1768557215.11 total 163.94 last 21.87) - open/close 3913 (time 1768557225.11 total 173.94 last 22.30) - open/close 4174 (time 1768557236.11 total 184.94 last 23.73) - open/close 4304 (time 1768557246.17 total 195.00 last 12.92) - open/close 4510 (time 1768557256.37 total 205.20 last 20.20) - open/close 4738 (time 1768557266.38 total 215.21 last 22.78) - open/close 4935 (time 1768557277.52 total 226.35 last 17.68) - open/close 5171 (time 1768557288.60 total 237.43 last 21.29) - open/close 5295 (time 1768557298.62 total 247.45 last 12.38) - open/close 5488 (time 1768557308.65 total 257.48 last 19.25) - open/close 5610 (time 1768557318.96 total 267.79 last 11.83) - open/close 5780 (time 1768557328.96 total 277.79 last 16.99) - open/close 5974 (time 1768557339.98 total 288.81 last 17.60) - open/close 6165 (time 1768557349.99 total 298.82 last 19.09) - open/close 6358 (time 1768557360.27 total 309.10 last 18.78) - open/close 6639 (time 1768557370.96 total 319.79 last 26.27) - open/close 6874 (time 1768557381.09 total 329.92 last 23.21) - open/close 7196 (time 1768557391.11 total 339.94 last 32.12) - open/close 7438 (time 1768557401.94 total 350.77 last 22.36) - open/close 7638 (time 1768557411.94 total 360.77 last 20.00) - open/close 7894 (time 1768557422.74 total 371.57 last 23.70) - open/close 8152 (time 1768557433.26 total 382.09 last 24.53) - open/close 8227 (time 1768557443.26 total 392.09 last 7.50) - open/close 8517 (time 1768557453.28 total 402.11 last 28.94) - open/close 8627 (time 1768557463.91 total 412.74 last 10.35) - open/close 8890 (time 1768557473.92 total 422.75 last 26.27) - open/close 9078 (time 1768557484.36 total 433.19 last 18.00) - open/close 9315 (time 1768557494.93 total 443.76 last 22.43) - open/close 9575 (time 1768557505.75 total 454.58 last 24.03) - open/close 9747 (time 1768557516.30 total 465.13 last 16.31) - open/close 9890 (time 1768557527.14 total 475.97 last 13.19) - open/close 10000 (time 1768557534.19 total 483.02 last 15.60) - open/close 10047 (time 1768557544.85 total 493.68 last 4.41) - open/close 10112 (time 1768557556.45 total 505.28 last 5.60) - open/close 10227 (time 1768557566.47 total 515.30 last 11.48) - open/close 10373 (time 1768557577.05 total 525.88 last 13.80) - open/close 10535 (time 1768557587.06 total 535.89 last 16.19) - open/close 10707 (time 1768557597.07 total 545.90 last 17.18) - open/close 10974 (time 1768557607.58 total 556.41 last 25.41) - open/close 11173 (time 1768557617.61 total 566.44 last 19.83) - open/close 11245 (time 1768557628.26 total 577.09 last 6.76) - open/close 11337 (time 1768557638.36 total 587.19 last 9.11) - open/close 11537 (time 1768557648.53 total 597.36 last 19.65) - open/close 11795 (time 1768557658.54 total 607.37 last 25.78) - open/close 11818 (time 1768557669.53 total 618.36 last 2.09) - open/close 11934 (time 1768557679.54 total 628.37 last 11.59) - open/close 12041 (time 1768557692.01 total 640.84 last 8.58) - open/close 12175 (time 1768557703.26 total 652.09 last 11.91) - open/close 12409 (time 1768557713.26 total 662.09 last 23.40) - open/close 12592 (time 1768557723.35 total 672.18 last 18.14) - open/close 12662 (time 1768557734.54 total 683.37 last 6.25) - open/close 12831 (time 1768557744.68 total 693.51 last 16.67) - open/close 13031 (time 1768557756.29 total 705.12 last 17.23) - open/close 13183 (time 1768557767.07 total 715.90 last 14.09) - open/close 13213 (time 1768557779.27 total 728.10 last 2.46) - open/close 13328 (time 1768557790.16 total 738.99 last 10.56) - open/close 13482 (time 1768557800.16 total 748.99 last 15.39) - open/close 13690 (time 1768557811.18 total 760.01 last 18.88) - open/close 13847 (time 1768557821.71 total 770.54 last 14.91) - open/close 13972 (time 1768557831.72 total 780.55 last 12.49) - open/close 14228 (time 1768557842.45 total 791.28 last 23.86) - open/close 14481 (time 1768557853.04 total 801.87 last 23.90) - open/close 14660 (time 1768557863.58 total 812.41 last 16.98) - open/close 14922 (time 1768557874.46 total 823.29 last 24.07) - open/close 15147 (time 1768557884.57 total 833.40 last 22.27) - open/close 15317 (time 1768557899.01 total 847.84 last 11.77) - open/close 15529 (time 1768557909.54 total 858.37 last 20.13) - open/close 15721 (time 1768557919.89 total 868.72 last 18.55) - open/close 15901 (time 1768557929.90 total 878.73 last 17.97) - open/close 16039 (time 1768557939.93 total 888.76 last 13.77) - open/close 16217 (time 1768557950.13 total 898.96 last 17.44) - open/close 16506 (time 1768557961.07 total 909.90 last 26.41) - open/close 16616 (time 1768557971.09 total 919.92 last 10.98) - open/close 16805 (time 1768557981.10 total 929.93 last 18.90) - open/close 17005 (time 1768557991.31 total 940.14 last 19.59) - open/close 17258 (time 1768558002.00 total 950.83 last 23.66) - open/close 17548 (time 1768558012.82 total 961.65 last 26.81) - open/close 17738 (time 1768558022.85 total 971.68 last 18.94) - open/close 17928 (time 1768558035.88 total 984.71 last 14.58) - open/close 18596 (time 1768558045.89 total 994.72 last 66.79) - open/close 19077 (time 1768558055.90 total 1004.73 last 48.06) - open/close 19844 (time 1768558065.91 total 1014.74 last 76.61) - open/close 20531 (time 1768558075.91 total 1024.74 last 68.67) - open/close 21224 (time 1768558085.92 total 1034.75 last 69.25) - open/close 22044 (time 1768558095.93 total 1044.76 last 81.94) - open/close 22847 (time 1768558105.93 total 1054.76 last 80.24) - open/close 23760 (time 1768558115.94 total 1064.77 last 91.26) - open/close 24705 (time 1768558125.94 total 1074.77 last 94.50) - open/close 25489 (time 1768558135.95 total 1084.78 last 78.34) - open/close 26311 (time 1768558145.95 total 1094.78 last 82.15) - open/close 27089 (time 1768558155.96 total 1104.79 last 77.73) - open/close 27829 (time 1768558165.98 total 1114.81 last 73.87) - open/close 28660 (time 1768558176.00 total 1124.83 last 82.93) - open/close 29221 (time 1768558186.03 total 1134.86 last 55.92) - open/close 29903 (time 1768558196.04 total 1144.87 last 68.16) - open/close 30684 (time 1768558206.04 total 1154.87 last 78.09) - open/close 31409 (time 1768558216.04 total 1164.87 last 72.50) - open/close 31917 (time 1768558226.05 total 1174.88 last 50.75) - open/close 32417 (time 1768558236.06 total 1184.89 last 49.95) - open/close 33250 (time 1768558246.07 total 1194.90 last 83.20) - open/close 34044 (time 1768558256.07 total 1204.90 last 79.37) - open/close 34950 (time 1768558266.07 total 1214.90 last 90.60) - open/close 35863 (time 1768558276.08 total 1224.91 last 91.26) - open/close 36791 (time 1768558286.14 total 1234.97 last 92.16) - open/close 37628 (time 1768558296.16 total 1244.99 last 83.60) - open/close 38490 (time 1768558306.17 total 1255.00 last 86.11) - open/close 39237 (time 1768558316.18 total 1265.01 last 74.57) - open/close 39944 (time 1768558326.19 total 1275.02 last 70.67) - open/close 40752 (time 1768558336.19 total 1285.02 last 80.76) - open/close 41712 (time 1768558346.20 total 1295.03 last 95.90) - open/close 42630 (time 1768558356.21 total 1305.04 last 91.77) - open/close 43687 (time 1768558366.22 total 1315.05 last 105.53) - open/close 44492 (time 1768558376.23 total 1325.06 last 80.48) - open/close 45383 (time 1768558386.23 total 1335.06 last 89.05) - open/close 46280 (time 1768558396.26 total 1345.09 last 89.46) - open/close 47167 (time 1768558406.27 total 1355.10 last 88.60) - open/close 47957 (time 1768558416.28 total 1365.11 last 78.93) - open/close 48778 (time 1768558426.29 total 1375.12 last 82.01) - open/close 49599 (time 1768558436.32 total 1385.15 last 81.87) total: 50000 open/close in 1391.32 seconds: 35.94 ops/second - unlinked 0 (time 1768558446 ; total 0 ; last 0) - unlinked 10000 (time 1768558587 ; total 141 ; last 141) - unlinked 20000 (time 1768558728 ; total 282 ; last 141) - unlinked 30000 (time 1768558871 ; total 425 ; last 143) - unlinked 40000 (time 1768559008 ; total 562 ; last 137) total: 50000 unlinks in 699 seconds: 71.530762 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg254-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg254-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg254-server stop mds service on oleg254-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg254-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing unload_modules_local oleg254-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg254-client: oleg254-client: ssh exited with exit code 2 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 2 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 2 pdsh@oleg254-client: oleg254-client: ssh exited with exit code 2