== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 04:18:46 (1787386726) start mds service on oleg425-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg425-server' oleg425-server: oleg425-server.virtnet: executing load_modules_local oleg425-server: Loading modules from /home/green/git/lustre-release/lustre oleg425-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg425-server: detected 4 online CPUs by sysfs oleg425-server: MODOPTS_LIBCFS= oleg425-server: Force libcfs to create 2 CPU partitions oleg425-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg425-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg425-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg425-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Started lustre-OST0000 oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg425-client.virtnet: -o user_xattr,flock 192.168.204.125@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 74 (time 1787386812.77 total 10.25 last 7.22) - open/close 242 (time 1787386822.78 total 20.26 last 16.78) - open/close 389 (time 1787386832.81 total 30.29 last 14.65) - open/close 541 (time 1787386843.17 total 40.65 last 14.67) - open/close 788 (time 1787386853.91 total 51.39 last 22.99) - open/close 1008 (time 1787386863.93 total 61.41 last 21.97) - open/close 1219 (time 1787386873.93 total 71.41 last 21.10) - open/close 1409 (time 1787386884.38 total 81.85 last 18.19) - open/close 1487 (time 1787386895.26 total 92.73 last 7.17) - open/close 1598 (time 1787386905.96 total 103.44 last 10.37) - open/close 1828 (time 1787386917.71 total 115.18 last 19.58) - open/close 2006 (time 1787386927.71 total 125.19 last 17.79) - open/close 2142 (time 1787386938.19 total 135.67 last 12.98) - open/close 2421 (time 1787386950.99 total 148.47 last 21.80) - open/close 2682 (time 1787386961.01 total 158.49 last 26.03) - open/close 2826 (time 1787386972.05 total 169.53 last 13.05) - open/close 2920 (time 1787386982.15 total 179.63 last 9.31) - open/close 3155 (time 1787386992.52 total 189.99 last 22.67) - open/close 3434 (time 1787387003.42 total 200.90 last 25.58) - open/close 3662 (time 1787387013.91 total 211.39 last 21.73) - open/close 3863 (time 1787387023.92 total 221.40 last 20.08) - open/close 4158 (time 1787387034.51 total 231.99 last 27.87) - open/close 4184 (time 1787387045.23 total 242.70 last 2.43) - open/close 4383 (time 1787387055.23 total 252.71 last 19.89) - open/close 4622 (time 1787387065.25 total 262.73 last 23.86) - open/close 4867 (time 1787387075.40 total 272.88 last 24.14) - open/close 5165 (time 1787387085.40 total 282.88 last 29.79) - open/close 5387 (time 1787387095.95 total 293.43 last 21.05) - open/close 5512 (time 1787387105.95 total 303.43 last 12.50) - open/close 5717 (time 1787387117.65 total 315.13 last 17.53) - open/close 5819 (time 1787387128.02 total 325.50 last 9.83) - open/close 5941 (time 1787387138.02 total 335.50 last 12.20) - open/close 6077 (time 1787387148.04 total 345.51 last 13.58) - open/close 6159 (time 1787387158.05 total 355.53 last 8.19) - open/close 6350 (time 1787387168.07 total 365.55 last 19.08) - open/close 6530 (time 1787387178.59 total 376.07 last 17.11) - open/close 6671 (time 1787387191.33 total 388.80 last 11.07) - open/close 6876 (time 1787387201.84 total 399.32 last 19.50) - open/close 7128 (time 1787387211.85 total 409.33 last 25.17) - open/close 7376 (time 1787387221.85 total 419.33 last 24.80) - open/close 7482 (time 1787387232.15 total 429.63 last 10.29) - open/close 7599 (time 1787387243.03 total 440.51 last 10.75) - open/close 7730 (time 1787387253.51 total 450.99 last 12.51) - open/close 7884 (time 1787387264.15 total 461.63 last 14.47) - open/close 8076 (time 1787387274.40 total 471.88 last 18.72) - open/close 8315 (time 1787387284.75 total 482.23 last 23.10) - open/close 8554 (time 1787387295.59 total 493.07 last 22.05) - open/close 8835 (time 1787387305.62 total 503.09 last 28.03) - open/close 8983 (time 1787387315.63 total 513.11 last 14.78) - open/close 9223 (time 1787387325.64 total 523.12 last 23.98) - open/close 9426 (time 1787387336.60 total 534.08 last 18.52) - open/close 9669 (time 1787387347.15 total 544.63 last 23.02) - open/close 9919 (time 1787387357.16 total 554.64 last 24.99) - open/close 10000 (time 1787387362.02 total 559.50 last 16.65) - open/close 10137 (time 1787387372.18 total 569.66 last 13.49) - open/close 10249 (time 1787387383.00 total 580.48 last 10.35) - open/close 10437 (time 1787387393.01 total 590.49 last 18.77) - open/close 10512 (time 1787387403.04 total 600.52 last 7.48) - open/close 10626 (time 1787387413.15 total 610.63 last 11.27) - open/close 10820 (time 1787387423.16 total 620.64 last 19.38) - open/close 11000 (time 1787387433.36 total 630.84 last 17.65) - open/close 11136 (time 1787387443.36 total 640.84 last 13.60) - open/close 11334 (time 1787387453.38 total 650.86 last 19.76) - open/close 11586 (time 1787387463.43 total 660.91 last 25.08) - open/close 11818 (time 1787387474.39 total 671.86 last 21.18) - open/close 12043 (time 1787387484.39 total 681.87 last 22.48) - open/close 12169 (time 1787387495.44 total 692.92 last 11.40) - open/close 12344 (time 1787387505.70 total 703.18 last 17.06) - open/close 12501 (time 1787387515.74 total 713.22 last 15.63) - open/close 12714 (time 1787387526.15 total 723.63 last 20.47) - open/close 12939 (time 1787387536.15 total 733.63 last 22.50) - open/close 13133 (time 1787387546.65 total 744.13 last 18.48) - open/close 13318 (time 1787387557.45 total 754.93 last 17.13) - open/close 13556 (time 1787387567.46 total 764.94 last 23.77) - open/close 13741 (time 1787387579.67 total 777.15 last 15.16) - open/close 13890 (time 1787387590.35 total 787.83 last 13.94) - open/close 14118 (time 1787387600.97 total 798.45 last 21.46) - open/close 14236 (time 1787387611.04 total 808.52 last 11.72) - open/close 14427 (time 1787387621.06 total 818.54 last 19.08) - open/close 14698 (time 1787387631.06 total 828.54 last 27.10) - open/close 14942 (time 1787387641.83 total 839.31 last 22.64) - open/close 15129 (time 1787387654.23 total 851.71 last 15.08) - open/close 15304 (time 1787387665.04 total 862.52 last 16.19) - open/close 15510 (time 1787387676.00 total 873.48 last 18.80) - open/close 15731 (time 1787387686.01 total 883.49 last 22.07) - open/close 15943 (time 1787387696.02 total 893.49 last 21.20) - open/close 16127 (time 1787387706.59 total 904.07 last 17.40) - open/close 16298 (time 1787387716.69 total 914.17 last 16.93) - open/close 16437 (time 1787387726.82 total 924.30 last 13.71) - open/close 16582 (time 1787387737.70 total 935.18 last 13.33) - open/close 16736 (time 1787387747.73 total 945.21 last 15.36) - open/close 16916 (time 1787387758.69 total 956.17 last 16.43) - open/close 17113 (time 1787387769.53 total 967.00 last 18.17) - open/close 17341 (time 1787387779.61 total 977.09 last 22.60) - open/close 17560 (time 1787387790.10 total 987.58 last 20.89) - open/close 17765 (time 1787387800.68 total 998.16 last 19.38) - open/close 17935 (time 1787387810.69 total 1008.16 last 16.99) - open/close 18260 (time 1787387820.69 total 1018.17 last 32.49) - open/close 18842 (time 1787387830.70 total 1028.18 last 58.15) - open/close 19443 (time 1787387840.70 total 1038.18 last 60.08) - open/close 20000 (time 1787387850.27 total 1047.75 last 58.23) - open/close 20685 (time 1787387860.29 total 1057.77 last 68.34) - open/close 21356 (time 1787387870.29 total 1067.77 last 67.08) - open/close 22160 (time 1787387880.30 total 1077.78 last 80.35) - open/close 22827 (time 1787387890.30 total 1087.78 last 66.68) - open/close 23546 (time 1787387900.31 total 1097.79 last 71.86) - open/close 24269 (time 1787387910.31 total 1107.79 last 72.27) - open/close 25009 (time 1787387920.32 total 1117.79 last 73.97) - open/close 25754 (time 1787387930.32 total 1127.80 last 74.47) - open/close 26480 (time 1787387940.34 total 1137.82 last 72.48) - open/close 27190 (time 1787387950.34 total 1147.82 last 70.95) - open/close 27862 (time 1787387960.35 total 1157.83 last 67.17) - open/close 28596 (time 1787387970.35 total 1167.83 last 73.39) - open/close 29246 (time 1787387980.36 total 1177.83 last 64.96) - open/close 29939 (time 1787387990.36 total 1187.84 last 69.24) - open/close 30625 (time 1787388000.39 total 1197.86 last 68.45) - open/close 31349 (time 1787388010.39 total 1207.87 last 72.35) - open/close 32076 (time 1787388020.39 total 1217.87 last 72.69) - open/close 32799 (time 1787388030.40 total 1227.87 last 72.29) - open/close 33513 (time 1787388040.41 total 1237.89 last 71.29) - open/close 34224 (time 1787388050.42 total 1247.90 last 71.04) - open/close 34938 (time 1787388060.42 total 1257.90 last 71.37) - open/close 35669 (time 1787388070.43 total 1267.91 last 73.06) - open/close 36369 (time 1787388080.44 total 1277.92 last 69.89) - open/close 37072 (time 1787388090.45 total 1287.93 last 70.27) - open/close 37786 (time 1787388100.48 total 1297.96 last 71.14) - open/close 38501 (time 1787388110.48 total 1307.96 last 71.50) - open/close 39207 (time 1787388120.49 total 1317.97 last 70.58) - open/close 39897 (time 1787388130.50 total 1327.98 last 68.91) - open/close 40591 (time 1787388140.52 total 1337.99 last 69.30) - open/close 41268 (time 1787388150.52 total 1348.00 last 67.68) - open/close 41925 (time 1787388160.53 total 1358.00 last 65.66) - open/close 42657 (time 1787388170.54 total 1368.01 last 73.13) - open/close 43386 (time 1787388180.54 total 1378.02 last 72.86) - open/close 44075 (time 1787388190.54 total 1388.02 last 68.88) - open/close 44692 (time 1787388200.55 total 1398.02 last 61.69) - open/close 45393 (time 1787388210.55 total 1408.03 last 70.08) - open/close 46052 (time 1787388220.57 total 1418.05 last 65.78) - open/close 46784 (time 1787388230.58 total 1428.06 last 73.12) - open/close 47409 (time 1787388240.58 total 1438.06 last 62.48) - open/close 48149 (time 1787388250.58 total 1448.06 last 73.99) - open/close 48868 (time 1787388260.61 total 1458.09 last 71.72) - open/close 49605 (time 1787388270.62 total 1468.10 last 73.62) total: 50000 open/close in 1473.27 seconds: 33.94 ops/second - unlinked 0 (time 1787388279 ; total 0 ; last 0) - unlinked 10000 (time 1787388426 ; total 147 ; last 147) - unlinked 20000 (time 1787388574 ; total 295 ; last 148) - unlinked 30000 (time 1787388721 ; total 442 ; last 147) - unlinked 40000 (time 1787388867 ; total 588 ; last 146) total: 50000 unlinks in 745 seconds: 67.114098 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg425-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg425-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg425-server stop mds service on oleg425-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg425-server unloading modules via unload_modules_local on: 'oleg425-server' oleg425-server: oleg425-server.virtnet: executing unload_modules_local oleg425-server: modules unloaded. pdsh@oleg425-client: oleg425-client: ssh exited with exit code 2 pdsh@oleg425-client: oleg425-server: ssh exited with exit code 2 pdsh@oleg425-client: oleg425-client: ssh exited with exit code 2 pdsh@oleg425-client: oleg425-server: ssh exited with exit code 2