== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 15:53:45 (1782244425) start mds service on oleg444-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg444-server' oleg444-server: oleg444-server.virtnet: executing load_modules_local oleg444-server: Loading modules from /home/green/git/lustre-release/lustre oleg444-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg444-server: detected 4 online CPUs by sysfs oleg444-server: MODOPTS_LIBCFS= oleg444-server: Force libcfs to create 2 CPU partitions oleg444-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg444-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg444-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg444-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-OST0000 oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg444-client.virtnet: -o user_xattr,flock 192.168.204.144@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 107 (time 1782244518.77 total 10.12 last 10.57) - open/close 302 (time 1782244529.78 total 21.14 last 17.70) - open/close 328 (time 1782244540.79 total 32.15 last 2.36) - open/close 371 (time 1782244553.38 total 44.74 last 3.41) - open/close 506 (time 1782244565.67 total 57.03 last 10.99) - open/close 654 (time 1782244575.99 total 67.35 last 14.34) - open/close 775 (time 1782244586.15 total 77.50 last 11.91) - open/close 833 (time 1782244597.09 total 88.45 last 5.30) - open/close 1010 (time 1782244607.10 total 98.46 last 17.69) - open/close 1230 (time 1782244617.11 total 108.47 last 21.96) - open/close 1392 (time 1782244627.13 total 118.49 last 16.17) - open/close 1459 (time 1782244637.38 total 128.74 last 6.54) - open/close 1554 (time 1782244648.45 total 139.81 last 8.58) - open/close 1685 (time 1782244661.18 total 152.54 last 10.30) - open/close 1788 (time 1782244672.45 total 163.81 last 9.14) - open/close 1925 (time 1782244683.26 total 174.61 last 12.68) - open/close 2060 (time 1782244693.27 total 184.62 last 13.49) - open/close 2146 (time 1782244703.92 total 195.27 last 8.08) - open/close 2246 (time 1782244714.48 total 205.84 last 9.46) - open/close 2360 (time 1782244725.45 total 216.81 last 10.39) - open/close 2535 (time 1782244735.46 total 226.81 last 17.49) - open/close 2707 (time 1782244746.35 total 237.70 last 15.79) - open/close 2884 (time 1782244756.36 total 247.72 last 17.67) - open/close 2957 (time 1782244768.25 total 259.61 last 6.14) - open/close 3068 (time 1782244778.27 total 269.63 last 11.08) - open/close 3141 (time 1782244788.47 total 279.83 last 7.16) - open/close 3291 (time 1782244799.47 total 290.83 last 13.63) - open/close 3455 (time 1782244809.48 total 300.83 last 16.39) - open/close 3568 (time 1782244819.57 total 310.93 last 11.20) - open/close 3722 (time 1782244829.59 total 320.95 last 15.37) - open/close 3801 (time 1782244839.60 total 330.96 last 7.89) - open/close 3992 (time 1782244850.02 total 341.38 last 18.33) - open/close 4201 (time 1782244860.41 total 351.76 last 20.13) - open/close 4297 (time 1782244872.62 total 363.98 last 7.86) - open/close 4497 (time 1782244883.58 total 374.93 last 18.26) - open/close 4598 (time 1782244893.58 total 384.94 last 10.10) - open/close 4740 (time 1782244903.59 total 394.95 last 14.18) - open/close 4836 (time 1782244914.49 total 405.85 last 8.81) - open/close 4905 (time 1782244924.52 total 415.88 last 6.88) - open/close 5115 (time 1782244934.53 total 425.89 last 20.98) - open/close 5236 (time 1782244946.36 total 437.72 last 10.22) - open/close 5427 (time 1782244956.49 total 447.84 last 18.87) - open/close 5631 (time 1782244967.34 total 458.70 last 18.79) - open/close 5824 (time 1782244977.83 total 469.19 last 18.40) - open/close 5990 (time 1782244987.83 total 479.19 last 16.59) - open/close 6065 (time 1782244997.88 total 489.24 last 7.47) - open/close 6270 (time 1782245007.88 total 499.24 last 20.50) - open/close 6484 (time 1782245017.90 total 509.26 last 21.37) - open/close 6652 (time 1782245028.68 total 520.03 last 15.59) - open/close 6814 (time 1782245039.09 total 530.44 last 15.56) - open/close 7028 (time 1782245049.10 total 540.46 last 21.37) - open/close 7246 (time 1782245059.15 total 550.51 last 21.69) - open/close 7402 (time 1782245071.01 total 562.37 last 13.15) - open/close 7644 (time 1782245083.06 total 574.42 last 20.08) - open/close 7813 (time 1782245096.27 total 587.63 last 12.80) - open/close 8027 (time 1782245109.30 total 600.65 last 16.43) - open/close 8186 (time 1782245119.30 total 610.66 last 15.88) - open/close 8303 (time 1782245129.99 total 621.34 last 10.95) - open/close 8483 (time 1782245140.97 total 632.33 last 16.38) - open/close 8592 (time 1782245151.54 total 642.90 last 10.32) - open/close 8776 (time 1782245161.89 total 653.25 last 17.78) - open/close 8978 (time 1782245171.91 total 663.27 last 20.16) - open/close 9202 (time 1782245182.50 total 673.86 last 21.14) - open/close 9438 (time 1782245192.52 total 683.87 last 23.57) - open/close 9548 (time 1782245203.31 total 694.67 last 10.19) - open/close 9638 (time 1782245218.23 total 709.59 last 6.03) - open/close 9811 (time 1782245228.24 total 719.60 last 17.28) - open/close 9908 (time 1782245238.25 total 729.61 last 9.69) - open/close 10000 (time 1782245248.07 total 739.43 last 9.37) - open/close 10157 (time 1782245259.02 total 750.38 last 14.34) - open/close 10262 (time 1782245269.20 total 760.56 last 10.31) - open/close 10298 (time 1782245279.73 total 771.09 last 3.42) - open/close 10415 (time 1782245289.75 total 781.11 last 11.68) - open/close 10573 (time 1782245299.81 total 791.17 last 15.70) - open/close 10640 (time 1782245309.84 total 801.20 last 6.68) - open/close 10802 (time 1782245320.71 total 812.07 last 14.91) - open/close 10947 (time 1782245330.73 total 822.09 last 14.47) - open/close 11052 (time 1782245340.75 total 832.11 last 10.48) - open/close 11110 (time 1782245351.47 total 842.83 last 5.41) - open/close 11233 (time 1782245362.92 total 854.27 last 10.75) - open/close 11408 (time 1782245373.50 total 864.86 last 16.53) - open/close 11527 (time 1782245384.81 total 876.17 last 10.52) - open/close 11603 (time 1782245395.37 total 886.73 last 7.20) - open/close 11698 (time 1782245405.40 total 896.76 last 9.47) - open/close 11752 (time 1782245415.94 total 907.30 last 5.12) - open/close 11922 (time 1782245425.94 total 917.30 last 16.99) - open/close 12164 (time 1782245436.90 total 928.26 last 22.08) - open/close 12291 (time 1782245447.08 total 938.44 last 12.47) - open/close 12404 (time 1782245457.10 total 948.46 last 11.28) - open/close 12536 (time 1782245467.77 total 959.12 last 12.38) - open/close 12719 (time 1782245480.44 total 971.80 last 14.44) - open/close 12820 (time 1782245491.83 total 983.19 last 8.87) - open/close 13052 (time 1782245501.84 total 993.20 last 23.18) - open/close 13104 (time 1782245513.41 total 1004.77 last 4.49) - open/close 13192 (time 1782245524.01 total 1015.37 last 8.30) - open/close 13364 (time 1782245534.04 total 1025.40 last 17.15) - open/close 13512 (time 1782245544.61 total 1035.97 last 14.01) - open/close 13649 (time 1782245555.52 total 1046.88 last 12.56) - open/close 13738 (time 1782245569.15 total 1060.51 last 6.53) - open/close 13884 (time 1782245579.33 total 1070.69 last 14.34) - open/close 14051 (time 1782245589.33 total 1080.69 last 16.70) - open/close 14586 (time 1782245599.35 total 1090.71 last 53.41) - open/close 15172 (time 1782245609.35 total 1100.71 last 58.58) - open/close 15702 (time 1782245619.37 total 1110.72 last 52.94) - open/close 16284 (time 1782245629.37 total 1120.73 last 58.17) - open/close 16832 (time 1782245639.37 total 1130.73 last 54.77) - open/close 17396 (time 1782245649.39 total 1140.75 last 56.32) - open/close 17945 (time 1782245659.41 total 1150.77 last 54.80) - open/close 18492 (time 1782245669.42 total 1160.77 last 54.65) - open/close 19105 (time 1782245679.43 total 1170.79 last 61.23) - open/close 19729 (time 1782245689.44 total 1180.80 last 62.30) - open/close 20000 (time 1782245693.91 total 1185.27 last 60.63) - open/close 20598 (time 1782245703.92 total 1195.28 last 59.76) - open/close 21180 (time 1782245713.93 total 1205.29 last 58.15) - open/close 21760 (time 1782245723.94 total 1215.30 last 57.91) - open/close 22399 (time 1782245733.95 total 1225.31 last 63.86) - open/close 23019 (time 1782245743.96 total 1235.31 last 61.96) - open/close 23606 (time 1782245753.97 total 1245.33 last 58.63) - open/close 24238 (time 1782245763.98 total 1255.33 last 63.15) - open/close 24826 (time 1782245773.98 total 1265.34 last 58.77) - open/close 25461 (time 1782245783.99 total 1275.35 last 63.45) - open/close 25978 (time 1782245794.01 total 1285.37 last 51.58) - open/close 26457 (time 1782245804.05 total 1295.41 last 47.74) - open/close 27007 (time 1782245814.07 total 1305.43 last 54.86) - open/close 27579 (time 1782245824.08 total 1315.44 last 57.17) - open/close 28170 (time 1782245834.09 total 1325.45 last 59.03) - open/close 28722 (time 1782245844.10 total 1335.46 last 55.15) - open/close 29245 (time 1782245854.11 total 1345.46 last 52.26) - open/close 29828 (time 1782245864.13 total 1355.49 last 58.14) - open/close 30000 (time 1782245866.78 total 1358.14 last 64.95) - open/close 30476 (time 1782245876.79 total 1368.15 last 47.56) - open/close 30953 (time 1782245886.79 total 1378.15 last 47.69) - open/close 31540 (time 1782245896.79 total 1388.15 last 58.70) - open/close 32108 (time 1782245906.81 total 1398.17 last 56.70) - open/close 32636 (time 1782245916.83 total 1408.19 last 52.70) - open/close 33227 (time 1782245926.84 total 1418.19 last 59.06) - open/close 33833 (time 1782245936.85 total 1428.21 last 60.50) - open/close 34410 (time 1782245946.86 total 1438.22 last 57.67) - open/close 35030 (time 1782245956.88 total 1448.24 last 61.85) - open/close 35619 (time 1782245966.88 total 1458.24 last 58.88) - open/close 36175 (time 1782245976.90 total 1468.26 last 55.52) - open/close 36733 (time 1782245986.91 total 1478.26 last 55.77) - open/close 37278 (time 1782245996.92 total 1488.28 last 54.44) - open/close 37811 (time 1782246006.95 total 1498.30 last 53.15) - open/close 38426 (time 1782246016.96 total 1508.32 last 61.42) - open/close 39000 (time 1782246026.96 total 1518.32 last 57.36) - open/close 39539 (time 1782246036.97 total 1528.33 last 53.89) - open/close 40000 (time 1782246046.29 total 1537.65 last 49.42) - open/close 40493 (time 1782246056.31 total 1547.67 last 49.22) - open/close 41030 (time 1782246066.32 total 1557.67 last 53.68) - open/close 41667 (time 1782246076.33 total 1567.69 last 63.60) - open/close 42250 (time 1782246086.34 total 1577.70 last 58.26) - open/close 42818 (time 1782246096.34 total 1587.70 last 56.77) - open/close 43456 (time 1782246106.35 total 1597.71 last 63.77) - open/close 44048 (time 1782246116.35 total 1607.71 last 59.17) - open/close 44655 (time 1782246126.36 total 1617.71 last 60.68) - open/close 45202 (time 1782246136.37 total 1627.72 last 54.65) - open/close 45786 (time 1782246146.37 total 1637.72 last 58.39) - open/close 46337 (time 1782246156.37 total 1647.73 last 55.10) - open/close 46970 (time 1782246166.37 total 1657.73 last 63.29) - open/close 47549 (time 1782246176.37 total 1667.73 last 57.87) - open/close 48126 (time 1782246186.39 total 1677.75 last 57.60) - open/close 48673 (time 1782246196.39 total 1687.75 last 54.68) - open/close 49177 (time 1782246206.41 total 1697.76 last 50.33) - open/close 49796 (time 1782246216.42 total 1707.78 last 61.83) total: 50000 open/close in 1711.06 seconds: 29.22 ops/second - unlinked 0 (time 1782246223 ; total 0 ; last 0) - unlinked 10000 (time 1782246396 ; total 173 ; last 173) - unlinked 20000 (time 1782246576 ; total 353 ; last 180) - unlinked 30000 (time 1782246768 ; total 545 ; last 192) - unlinked 40000 (time 1782246958 ; total 735 ; last 190) total: 50000 unlinks in 917 seconds: 54.525627 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg444-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg444-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg444-server stop mds service on oleg444-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg444-server unloading modules via unload_modules_local on: 'oleg444-server' oleg444-server: oleg444-server.virtnet: executing unload_modules_local oleg444-server: modules unloaded. pdsh@oleg444-client: oleg444-server: ssh exited with exit code 2 pdsh@oleg444-client: oleg444-client: ssh exited with exit code 2 pdsh@oleg444-client: oleg444-client: ssh exited with exit code 2 pdsh@oleg444-client: oleg444-server: ssh exited with exit code 2