== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:16:30 (1787609790) start mds service on oleg439-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions oleg439-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg439-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg439-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg439-server Start mds2: mount -t lustre -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg439-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1 Started lustre-OST0000 oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 69 (time 1787609909.88 total 10.30 last 6.70) - open/close 249 (time 1787609920.44 total 20.86 last 17.05) - open/close 371 (time 1787609930.53 total 30.96 last 12.08) - open/close 538 (time 1787609941.37 total 41.79 last 15.42) - open/close 663 (time 1787609951.38 total 51.81 last 12.48) - open/close 789 (time 1787609961.38 total 61.81 last 12.60) - open/close 910 (time 1787609971.75 total 72.18 last 11.67) - open/close 1007 (time 1787609981.76 total 82.18 last 9.70) - open/close 1088 (time 1787609992.29 total 92.72 last 7.69) - open/close 1126 (time 1787610003.81 total 104.24 last 3.30) - open/close 1282 (time 1787610014.38 total 114.81 last 14.76) - open/close 1509 (time 1787610025.32 total 125.74 last 20.76) - open/close 1628 (time 1787610035.32 total 135.74 last 11.90) - open/close 1699 (time 1787610045.74 total 146.16 last 6.81) - open/close 1841 (time 1787610055.75 total 156.17 last 14.19) - open/close 1924 (time 1787610066.24 total 166.66 last 7.91) - open/close 2034 (time 1787610076.28 total 176.70 last 10.96) - open/close 2163 (time 1787610086.71 total 187.13 last 12.37) - open/close 2251 (time 1787610096.72 total 197.14 last 8.79) - open/close 2399 (time 1787610107.66 total 208.08 last 13.53) - open/close 2498 (time 1787610117.66 total 218.09 last 9.90) - open/close 2624 (time 1787610128.55 total 228.97 last 11.57) - open/close 2723 (time 1787610141.61 total 242.04 last 7.58) - open/close 2986 (time 1787610151.64 total 252.07 last 26.22) - open/close 3187 (time 1787610163.19 total 263.61 last 17.41) - open/close 3250 (time 1787610174.25 total 274.67 last 5.69) - open/close 3459 (time 1787610184.28 total 284.71 last 20.83) - open/close 3566 (time 1787610194.30 total 294.72 last 10.69) - open/close 3656 (time 1787610204.85 total 305.27 last 8.53) - open/close 3736 (time 1787610215.05 total 315.48 last 7.84) - open/close 3875 (time 1787610226.13 total 326.55 last 12.55) - open/close 4046 (time 1787610236.15 total 336.57 last 17.06) - open/close 4275 (time 1787610246.22 total 346.65 last 22.73) - open/close 4370 (time 1787610256.75 total 357.18 last 9.02) - open/close 4540 (time 1787610267.31 total 367.73 last 16.10) - open/close 4617 (time 1787610277.54 total 377.97 last 7.52) - open/close 4689 (time 1787610287.66 total 388.09 last 7.11) - open/close 4776 (time 1787610300.21 total 400.63 last 6.93) - open/close 4831 (time 1787610310.95 total 411.37 last 5.12) - open/close 4934 (time 1787610320.96 total 421.38 last 10.29) - open/close 5118 (time 1787610333.70 total 434.12 last 14.44) - open/close 5289 (time 1787610344.11 total 444.53 last 16.43) - open/close 5372 (time 1787610354.11 total 454.54 last 8.30) - open/close 5424 (time 1787610364.60 total 465.02 last 4.96) - open/close 5627 (time 1787610374.63 total 475.05 last 20.25) - open/close 5758 (time 1787610384.64 total 485.06 last 13.08) - open/close 5895 (time 1787610395.00 total 495.42 last 13.22) - open/close 6101 (time 1787610405.92 total 506.34 last 18.87) - open/close 6240 (time 1787610415.96 total 516.39 last 13.84) - open/close 6474 (time 1787610426.78 total 527.21 last 21.62) - open/close 6571 (time 1787610437.30 total 537.72 last 9.23) - open/close 6725 (time 1787610449.96 total 550.38 last 12.16) - open/close 6792 (time 1787610460.01 total 560.43 last 6.66) - open/close 6956 (time 1787610470.50 total 570.92 last 15.64) - open/close 7019 (time 1787610481.46 total 581.89 last 5.75) - open/close 7285 (time 1787610491.46 total 591.89 last 26.59) - open/close 7375 (time 1787610501.54 total 601.96 last 8.93) - open/close 7582 (time 1787610511.54 total 611.97 last 20.69) - open/close 7775 (time 1787610521.57 total 621.99 last 19.25) - open/close 7984 (time 1787610531.65 total 632.08 last 20.72) - open/close 8188 (time 1787610542.25 total 642.67 last 19.26) - open/close 8295 (time 1787610553.27 total 653.70 last 9.70) - open/close 8491 (time 1787610563.63 total 664.05 last 18.93) - open/close 8627 (time 1787610573.67 total 674.09 last 13.54) - open/close 8762 (time 1787610583.93 total 684.35 last 13.16) - open/close 9026 (time 1787610594.48 total 694.91 last 25.01) - open/close 9250 (time 1787610604.51 total 704.93 last 22.35) - open/close 9456 (time 1787610615.48 total 715.91 last 18.77) - open/close 9618 (time 1787610625.49 total 725.92 last 16.18) - open/close 9752 (time 1787610636.47 total 736.89 last 12.21) - open/close 9892 (time 1787610646.50 total 746.92 last 13.96) - open/close 10000 (time 1787610651.09 total 751.52 last 23.49) - open/close 10148 (time 1787610661.80 total 762.22 last 13.83) - open/close 10261 (time 1787610674.31 total 774.74 last 9.03) - open/close 10393 (time 1787610684.46 total 784.88 last 13.02) - open/close 10439 (time 1787610694.95 total 795.38 last 4.38) - open/close 10673 (time 1787610705.84 total 806.26 last 21.50) - open/close 10911 (time 1787610718.64 total 819.07 last 18.58) - open/close 11056 (time 1787610729.25 total 829.67 last 13.67) - open/close 11231 (time 1787610739.86 total 840.28 last 16.49) - open/close 11408 (time 1787610749.88 total 850.30 last 17.67) - open/close 11631 (time 1787610759.91 total 860.34 last 22.22) - open/close 11882 (time 1787610770.88 total 871.31 last 22.88) - open/close 12023 (time 1787610785.33 total 885.75 last 9.76) - open/close 12163 (time 1787610795.83 total 896.25 last 13.33) - open/close 12341 (time 1787610806.06 total 906.48 last 17.40) - open/close 12526 (time 1787610816.07 total 916.49 last 18.48) - open/close 12747 (time 1787610826.12 total 926.54 last 21.99) - open/close 12801 (time 1787610836.15 total 936.58 last 5.38) - open/close 12871 (time 1787610848.25 total 948.67 last 5.79) - open/close 12926 (time 1787610858.35 total 958.77 last 5.45) - open/close 13048 (time 1787610868.35 total 968.78 last 12.19) - open/close 13259 (time 1787610878.37 total 978.80 last 21.06) - open/close 13502 (time 1787610889.37 total 989.80 last 22.09) - open/close 13725 (time 1787610899.38 total 999.80 last 22.29) - open/close 13851 (time 1787610909.69 total 1010.11 last 12.22) - open/close 14034 (time 1787610920.12 total 1020.54 last 17.54) - open/close 14161 (time 1787610930.44 total 1030.86 last 12.31) - open/close 14353 (time 1787610940.77 total 1041.20 last 18.58) - open/close 14447 (time 1787610951.17 total 1051.60 last 9.04) - open/close 14522 (time 1787610961.58 total 1062.01 last 7.21) - open/close 14578 (time 1787610971.72 total 1072.14 last 5.53) - open/close 14682 (time 1787610982.02 total 1082.45 last 10.09) - open/close 14786 (time 1787610992.05 total 1092.47 last 10.37) - open/close 14911 (time 1787611003.75 total 1104.18 last 10.68) - open/close 15038 (time 1787611013.76 total 1114.19 last 12.68) - open/close 15605 (time 1787611023.77 total 1124.19 last 56.67) - open/close 16204 (time 1787611033.77 total 1134.20 last 59.87) - open/close 16915 (time 1787611043.78 total 1144.20 last 71.07) - open/close 17531 (time 1787611053.78 total 1154.21 last 61.59) - open/close 18167 (time 1787611063.79 total 1164.21 last 63.56) - open/close 18792 (time 1787611073.79 total 1174.21 last 62.50) - open/close 19402 (time 1787611083.81 total 1184.23 last 60.90) - open/close 19955 (time 1787611093.81 total 1194.23 last 55.28) - open/close 20619 (time 1787611103.82 total 1204.24 last 66.35) - open/close 21259 (time 1787611113.82 total 1214.24 last 63.98) - open/close 21828 (time 1787611123.83 total 1224.25 last 56.85) - open/close 22319 (time 1787611133.85 total 1234.27 last 49.01) - open/close 22866 (time 1787611143.85 total 1244.27 last 54.69) - open/close 23452 (time 1787611153.85 total 1254.28 last 58.59) - open/close 24072 (time 1787611163.85 total 1264.28 last 61.99) - open/close 24674 (time 1787611173.86 total 1274.28 last 60.17) - open/close 25272 (time 1787611183.87 total 1284.30 last 59.70) - open/close 25877 (time 1787611193.88 total 1294.30 last 60.48) - open/close 26420 (time 1787611203.88 total 1304.31 last 54.26) - open/close 27060 (time 1787611213.89 total 1314.32 last 63.94) - open/close 27650 (time 1787611223.90 total 1324.33 last 58.95) - open/close 28307 (time 1787611233.91 total 1334.33 last 65.68) - open/close 28939 (time 1787611243.92 total 1344.34 last 63.13) - open/close 29515 (time 1787611253.94 total 1354.36 last 57.47) - open/close 30000 (time 1787611261.29 total 1361.72 last 65.95) - open/close 30623 (time 1787611271.30 total 1371.73 last 62.23) - open/close 31335 (time 1787611281.31 total 1381.73 last 71.18) - open/close 32018 (time 1787611291.32 total 1391.74 last 68.21) - open/close 32690 (time 1787611301.33 total 1401.75 last 67.15) - open/close 33350 (time 1787611311.33 total 1411.75 last 66.00) - open/close 33949 (time 1787611321.34 total 1421.76 last 59.82) - open/close 34453 (time 1787611331.36 total 1431.78 last 50.30) - open/close 35126 (time 1787611341.37 total 1441.79 last 67.26) - open/close 35748 (time 1787611351.37 total 1451.79 last 62.17) - open/close 36238 (time 1787611361.39 total 1461.81 last 48.90) - open/close 36696 (time 1787611371.40 total 1471.82 last 45.76) - open/close 37283 (time 1787611381.41 total 1481.83 last 58.67) - open/close 37916 (time 1787611391.41 total 1491.84 last 63.25) - open/close 38528 (time 1787611401.42 total 1501.85 last 61.14) - open/close 39222 (time 1787611411.43 total 1511.85 last 69.35) - open/close 39893 (time 1787611421.44 total 1521.86 last 67.04) - open/close 40437 (time 1787611431.46 total 1531.89 last 54.27) - open/close 41066 (time 1787611441.46 total 1541.89 last 62.88) - open/close 41641 (time 1787611451.47 total 1551.89 last 57.47) - open/close 42203 (time 1787611461.51 total 1561.93 last 56.00) - open/close 42761 (time 1787611471.51 total 1571.94 last 55.77) - open/close 43421 (time 1787611481.55 total 1581.97 last 65.75) - open/close 44038 (time 1787611491.55 total 1591.98 last 61.67) - open/close 44622 (time 1787611501.56 total 1601.98 last 58.38) - open/close 45134 (time 1787611511.57 total 1611.99 last 51.17) - open/close 45639 (time 1787611521.57 total 1621.99 last 50.48) - open/close 46142 (time 1787611531.58 total 1632.01 last 50.22) - open/close 46688 (time 1787611541.60 total 1642.02 last 54.54) - open/close 47262 (time 1787611551.65 total 1652.08 last 57.08) - open/close 47810 (time 1787611561.65 total 1662.08 last 54.79) - open/close 48389 (time 1787611571.66 total 1672.08 last 57.89) - open/close 48922 (time 1787611581.67 total 1682.10 last 53.20) - open/close 49434 (time 1787611591.68 total 1692.11 last 51.15) total: 50000 open/close in 1701.42 seconds: 29.39 ops/second - unlinked 0 (time 1787611606 ; total 0 ; last 0) - unlinked 10000 (time 1787611783 ; total 177 ; last 177) - unlinked 20000 (time 1787611951 ; total 345 ; last 168) - unlinked 30000 (time 1787612122 ; total 516 ; last 171) - unlinked 40000 (time 1787612292 ; total 686 ; last 170) total: 50000 unlinks in 854 seconds: 58.548008 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg439-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg439-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg439-server stop mds service on oleg439-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg439-server stop mds service on oleg439-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg439-server unloading modules via unload_modules_local on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing unload_modules_local oleg439-server: modules unloaded. pdsh@oleg439-client: oleg439-client: ssh exited with exit code 2 pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2 pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2 pdsh@oleg439-client: oleg439-client: ssh exited with exit code 2