== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:20:27 (1788445227) start mds service on oleg108-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg108-server' oleg108-server: oleg108-server.virtnet: executing load_modules_local oleg108-server: Loading modules from /home/green/git/lustre-release/lustre oleg108-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg108-server: detected 4 online CPUs by sysfs oleg108-server: MODOPTS_LIBCFS= oleg108-server: Force libcfs to create 2 CPU partitions oleg108-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg108-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg108-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg108-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-OST0000 oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg108-client.virtnet: -o user_xattr,flock 192.168.201.108@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 230 (time 1788445323.76 total 10.54 last 21.83) - open/close 458 (time 1788445333.78 total 20.56 last 22.75) - open/close 734 (time 1788445344.11 total 30.88 last 26.73) - open/close 933 (time 1788445357.07 total 43.85 last 15.35) - open/close 1017 (time 1788445367.86 total 54.64 last 7.79) - open/close 1155 (time 1788445377.86 total 64.64 last 13.79) - open/close 1327 (time 1788445388.82 total 75.60 last 15.69) - open/close 1420 (time 1788445399.76 total 86.54 last 8.50) - open/close 1562 (time 1788445409.78 total 96.56 last 14.17) - open/close 1825 (time 1788445419.90 total 106.68 last 25.99) - open/close 2068 (time 1788445430.14 total 116.92 last 23.73) - open/close 2322 (time 1788445440.72 total 127.50 last 24.01) - open/close 2502 (time 1788445451.73 total 138.50 last 16.35) - open/close 2745 (time 1788445461.74 total 148.51 last 24.28) - open/close 2835 (time 1788445472.64 total 159.42 last 8.25) - open/close 2962 (time 1788445482.70 total 169.48 last 12.62) - open/close 3107 (time 1788445492.73 total 179.51 last 14.46) - open/close 3372 (time 1788445502.74 total 189.51 last 26.48) - open/close 3630 (time 1788445513.32 total 200.10 last 24.37) - open/close 3867 (time 1788445523.76 total 210.54 last 22.71) - open/close 4074 (time 1788445535.26 total 222.04 last 18.00) - open/close 4353 (time 1788445545.68 total 232.46 last 26.77) - open/close 4629 (time 1788445555.69 total 242.46 last 27.59) - open/close 4845 (time 1788445566.29 total 253.06 last 20.38) - open/close 4997 (time 1788445576.29 total 263.06 last 15.20) - open/close 5118 (time 1788445586.34 total 273.11 last 12.04) - open/close 5301 (time 1788445596.55 total 283.32 last 17.92) - open/close 5476 (time 1788445606.99 total 293.76 last 16.76) - open/close 5726 (time 1788445617.53 total 304.30 last 23.71) - open/close 5918 (time 1788445628.11 total 314.88 last 18.15) - open/close 6165 (time 1788445638.40 total 325.17 last 24.00) - open/close 6393 (time 1788445648.76 total 335.54 last 22.00) - open/close 6607 (time 1788445658.76 total 345.54 last 21.39) - open/close 6868 (time 1788445671.73 total 358.51 last 20.13) - open/close 7073 (time 1788445682.42 total 369.20 last 19.17) - open/close 7287 (time 1788445693.21 total 379.99 last 19.84) - open/close 7463 (time 1788445703.21 total 389.99 last 17.59) - open/close 7520 (time 1788445714.23 total 401.01 last 5.17) - open/close 7697 (time 1788445724.30 total 411.07 last 17.58) - open/close 7899 (time 1788445734.69 total 421.46 last 19.44) - open/close 8173 (time 1788445747.49 total 434.27 last 21.40) - open/close 8379 (time 1788445757.49 total 444.27 last 20.59) - open/close 8624 (time 1788445768.08 total 454.85 last 23.15) - open/close 8864 (time 1788445778.64 total 465.41 last 22.72) - open/close 8976 (time 1788445788.75 total 475.52 last 11.08) - open/close 9208 (time 1788445799.04 total 485.81 last 22.55) - open/close 9474 (time 1788445809.68 total 496.46 last 24.99) - open/close 9698 (time 1788445820.08 total 506.86 last 21.54) - open/close 9884 (time 1788445830.11 total 516.89 last 18.55) - open/close 10000 (time 1788445835.39 total 522.17 last 21.97) - open/close 10212 (time 1788445845.52 total 532.30 last 20.93) - open/close 10430 (time 1788445855.64 total 542.42 last 21.54) - open/close 10689 (time 1788445866.35 total 553.13 last 24.17) - open/close 10889 (time 1788445876.36 total 563.13 last 20.00) - open/close 11148 (time 1788445886.65 total 573.43 last 25.16) - open/close 11360 (time 1788445896.79 total 583.57 last 20.91) - open/close 11601 (time 1788445906.80 total 593.57 last 24.09) - open/close 11734 (time 1788445916.93 total 603.71 last 13.13) - open/close 12017 (time 1788445927.69 total 614.47 last 26.30) - open/close 12237 (time 1788445937.71 total 624.49 last 21.95) - open/close 12490 (time 1788445947.73 total 634.50 last 25.27) - open/close 12632 (time 1788445958.80 total 645.57 last 12.83) - open/close 12968 (time 1788445968.83 total 655.61 last 33.49) - open/close 13224 (time 1788445979.35 total 666.13 last 24.33) - open/close 13407 (time 1788445989.37 total 676.14 last 18.27) - open/close 13627 (time 1788446000.19 total 686.97 last 20.33) - open/close 13843 (time 1788446010.20 total 696.98 last 21.57) - open/close 14003 (time 1788446020.54 total 707.31 last 15.48) - open/close 14255 (time 1788446030.61 total 717.38 last 25.03) - open/close 14486 (time 1788446041.09 total 727.87 last 22.04) - open/close 14610 (time 1788446051.90 total 738.67 last 11.47) - open/close 14759 (time 1788446062.52 total 749.29 last 14.03) - open/close 14831 (time 1788446074.55 total 761.33 last 5.98) - open/close 14960 (time 1788446085.51 total 772.29 last 11.77) - open/close 15231 (time 1788446095.53 total 782.31 last 27.04) - open/close 15469 (time 1788446105.98 total 792.76 last 22.78) - open/close 15762 (time 1788446115.98 total 802.76 last 29.30) - open/close 15911 (time 1788446127.42 total 814.19 last 13.03) - open/close 16123 (time 1788446137.42 total 824.20 last 21.19) - open/close 16374 (time 1788446147.42 total 834.20 last 25.09) - open/close 16598 (time 1788446157.49 total 844.27 last 22.25) - open/close 16821 (time 1788446167.50 total 854.28 last 22.28) - open/close 17004 (time 1788446177.50 total 864.28 last 18.29) - open/close 17217 (time 1788446187.51 total 874.29 last 21.28) - open/close 17372 (time 1788446197.63 total 884.41 last 15.32) - open/close 17599 (time 1788446207.65 total 894.43 last 22.65) - open/close 17838 (time 1788446217.65 total 904.43 last 23.90) - open/close 18064 (time 1788446227.72 total 914.49 last 22.46) - open/close 18309 (time 1788446237.82 total 924.59 last 24.26) - open/close 18554 (time 1788446247.82 total 934.60 last 24.49) - open/close 18808 (time 1788446258.35 total 945.13 last 24.11) - open/close 19315 (time 1788446268.36 total 955.14 last 50.67) - open/close 20000 (time 1788446275.90 total 962.67 last 90.88) - open/close 20878 (time 1788446285.92 total 972.70 last 87.59) - open/close 21623 (time 1788446295.93 total 982.70 last 74.45) - open/close 22438 (time 1788446305.94 total 992.71 last 81.43) - open/close 23373 (time 1788446315.94 total 1002.72 last 93.47) - open/close 24386 (time 1788446325.94 total 1012.72 last 101.29) - open/close 25301 (time 1788446335.94 total 1022.72 last 91.46) - open/close 26246 (time 1788446345.95 total 1032.73 last 94.46) - open/close 27183 (time 1788446355.96 total 1042.73 last 93.63) - open/close 27882 (time 1788446365.96 total 1052.73 last 69.89) - open/close 28819 (time 1788446375.98 total 1062.75 last 93.53) - open/close 29743 (time 1788446385.98 total 1072.76 last 92.33) - open/close 30000 (time 1788446388.38 total 1075.16 last 106.98) - open/close 30984 (time 1788446398.39 total 1085.16 last 98.39) - open/close 31925 (time 1788446408.39 total 1095.17 last 94.02) - open/close 32612 (time 1788446418.42 total 1105.20 last 68.50) - open/close 33253 (time 1788446428.43 total 1115.21 last 64.05) - open/close 34225 (time 1788446438.43 total 1125.21 last 97.20) - open/close 35277 (time 1788446448.45 total 1135.23 last 104.98) - open/close 36222 (time 1788446458.46 total 1145.24 last 94.42) - open/close 37245 (time 1788446468.47 total 1155.25 last 102.19) - open/close 38249 (time 1788446478.48 total 1165.25 last 100.36) - open/close 39179 (time 1788446488.48 total 1175.25 last 92.98) - open/close 40000 (time 1788446497.88 total 1184.65 last 87.34) - open/close 40894 (time 1788446507.88 total 1194.66 last 89.37) - open/close 41848 (time 1788446517.89 total 1204.67 last 95.33) - open/close 42806 (time 1788446527.89 total 1214.67 last 95.76) - open/close 43733 (time 1788446537.90 total 1224.67 last 92.67) - open/close 44727 (time 1788446547.90 total 1234.67 last 99.38) - open/close 45722 (time 1788446557.90 total 1244.68 last 99.48) - open/close 46617 (time 1788446567.91 total 1254.69 last 89.41) - open/close 47582 (time 1788446577.92 total 1264.69 last 96.44) - open/close 48511 (time 1788446587.92 total 1274.70 last 92.82) - open/close 49423 (time 1788446597.92 total 1284.70 last 91.20) total: 50000 open/close in 1290.48 seconds: 38.75 ops/second - unlinked 0 (time 1788446607 ; total 0 ; last 0) - unlinked 10000 (time 1788446723 ; total 116 ; last 116) - unlinked 20000 (time 1788446821 ; total 214 ; last 98) - unlinked 30000 (time 1788446888 ; total 281 ; last 67) - unlinked 40000 (time 1788446954 ; total 347 ; last 66) total: 50000 unlinks in 411 seconds: 121.654503 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg108-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg108-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg108-server stop mds service on oleg108-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg108-server unloading modules via unload_modules_local on: 'oleg108-server' oleg108-server: oleg108-server.virtnet: executing unload_modules_local oleg108-server: modules unloaded. pdsh@oleg108-client: oleg108-client: ssh exited with exit code 2 pdsh@oleg108-client: oleg108-server: ssh exited with exit code 2 pdsh@oleg108-client: oleg108-client: ssh exited with exit code 2 pdsh@oleg108-client: oleg108-server: ssh exited with exit code 2