== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 11:20:24 (1788794424) start mds service on oleg412-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg412-server' oleg412-server: oleg412-server.virtnet: executing load_modules_local oleg412-server: Loading modules from /home/green/git/lustre-release/lustre oleg412-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg412-server: detected 4 online CPUs by sysfs oleg412-server: MODOPTS_LIBCFS= oleg412-server: Force libcfs to create 2 CPU partitions oleg412-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg412-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg412-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg412-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Started lustre-OST0000 oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg412-client.virtnet: -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 88 (time 1788794517.53 total 10.01 last 8.79) - open/close 198 (time 1788794527.53 total 20.02 last 11.00) - open/close 386 (time 1788794537.76 total 30.25 last 18.38) - open/close 642 (time 1788794548.81 total 41.30 last 23.16) - open/close 865 (time 1788794559.50 total 51.99 last 20.86) - open/close 1087 (time 1788794570.01 total 62.49 last 21.14) - open/close 1279 (time 1788794581.02 total 73.51 last 17.43) - open/close 1507 (time 1788794591.05 total 83.54 last 22.73) - open/close 1670 (time 1788794601.09 total 93.57 last 16.25) - open/close 1828 (time 1788794611.10 total 103.58 last 15.79) - open/close 1907 (time 1788794621.10 total 113.59 last 7.90) - open/close 2062 (time 1788794631.71 total 124.20 last 14.60) - open/close 2277 (time 1788794641.72 total 134.21 last 21.49) - open/close 2506 (time 1788794652.04 total 144.53 last 22.18) - open/close 2558 (time 1788794662.49 total 154.98 last 4.98) - open/close 2751 (time 1788794673.11 total 165.60 last 18.18) - open/close 2914 (time 1788794684.04 total 176.53 last 14.91) - open/close 3082 (time 1788794694.89 total 187.37 last 15.49) - open/close 3263 (time 1788794704.89 total 197.38 last 18.10) - open/close 3403 (time 1788794715.35 total 207.84 last 13.38) - open/close 3644 (time 1788794725.36 total 217.84 last 24.09) - open/close 3849 (time 1788794735.37 total 227.85 last 20.48) - open/close 4074 (time 1788794745.94 total 238.43 last 21.27) - open/close 4216 (time 1788794756.23 total 248.72 last 13.80) - open/close 4325 (time 1788794767.20 total 259.69 last 9.94) - open/close 4514 (time 1788794777.21 total 269.69 last 18.89) - open/close 4645 (time 1788794787.49 total 279.98 last 12.74) - open/close 4923 (time 1788794797.49 total 289.98 last 27.79) - open/close 5072 (time 1788794807.64 total 300.13 last 14.69) - open/close 5265 (time 1788794817.64 total 310.13 last 19.30) - open/close 5492 (time 1788794827.65 total 320.13 last 22.69) - open/close 5633 (time 1788794837.68 total 330.16 last 14.06) - open/close 5819 (time 1788794849.68 total 342.17 last 15.49) - open/close 5943 (time 1788794859.98 total 352.47 last 12.04) - open/close 6063 (time 1788794870.10 total 362.58 last 11.87) - open/close 6230 (time 1788794880.13 total 372.62 last 16.64) - open/close 6413 (time 1788794890.70 total 383.19 last 17.31) - open/close 6660 (time 1788794900.71 total 393.20 last 24.67) - open/close 6808 (time 1788794910.72 total 403.20 last 14.79) - open/close 7134 (time 1788794920.72 total 413.21 last 32.57) - open/close 7312 (time 1788794931.70 total 424.18 last 16.22) - open/close 7457 (time 1788794942.17 total 434.66 last 13.85) - open/close 7604 (time 1788794953.01 total 445.50 last 13.55) - open/close 7690 (time 1788794963.47 total 455.96 last 8.23) - open/close 7800 (time 1788794973.48 total 465.96 last 10.99) - open/close 8021 (time 1788794984.29 total 476.77 last 20.45) - open/close 8179 (time 1788794994.52 total 487.01 last 15.44) - open/close 8227 (time 1788795005.27 total 497.76 last 4.47) - open/close 8314 (time 1788795015.95 total 508.44 last 8.15) - open/close 8404 (time 1788795026.72 total 519.20 last 8.36) - open/close 8585 (time 1788795037.39 total 529.88 last 16.95) - open/close 8705 (time 1788795047.88 total 540.36 last 11.44) - open/close 8814 (time 1788795057.89 total 550.37 last 10.89) - open/close 8906 (time 1788795068.75 total 561.23 last 8.47) - open/close 9088 (time 1788795079.64 total 572.13 last 16.70) - open/close 9309 (time 1788795092.69 total 585.18 last 16.94) - open/close 9542 (time 1788795102.69 total 595.18 last 23.29) - open/close 9730 (time 1788795113.62 total 606.11 last 17.21) - open/close 9994 (time 1788795124.38 total 616.86 last 24.54) - open/close 10216 (time 1788795134.39 total 626.88 last 22.17) - open/close 10336 (time 1788795144.68 total 637.17 last 11.66) - open/close 10485 (time 1788795154.71 total 647.20 last 14.86) - open/close 10545 (time 1788795164.81 total 657.30 last 5.94) - open/close 10697 (time 1788795175.63 total 668.12 last 14.05) - open/close 10933 (time 1788795185.76 total 678.25 last 23.30) - open/close 11157 (time 1788795196.02 total 688.51 last 21.83) - open/close 11374 (time 1788795206.02 total 698.51 last 21.70) - open/close 11592 (time 1788795216.05 total 708.53 last 21.75) - open/close 11722 (time 1788795226.06 total 718.55 last 12.98) - open/close 11953 (time 1788795236.82 total 729.30 last 21.48) - open/close 12215 (time 1788795246.82 total 739.31 last 26.19) - open/close 12461 (time 1788795256.84 total 749.33 last 24.55) - open/close 12629 (time 1788795266.86 total 759.34 last 16.78) - open/close 12800 (time 1788795277.06 total 769.55 last 16.75) - open/close 12968 (time 1788795287.08 total 779.56 last 16.78) - open/close 12999 (time 1788795297.19 total 789.68 last 3.06) - open/close 13186 (time 1788795309.22 total 801.70 last 15.55) - open/close 13454 (time 1788795319.22 total 811.71 last 26.79) - open/close 13641 (time 1788795329.23 total 821.72 last 18.67) - open/close 13792 (time 1788795339.25 total 831.74 last 15.07) - open/close 13907 (time 1788795349.61 total 842.10 last 11.11) - open/close 14006 (time 1788795359.62 total 852.10 last 9.89) - open/close 14178 (time 1788795369.63 total 862.12 last 17.17) - open/close 14289 (time 1788795380.06 total 872.55 last 10.64) - open/close 14483 (time 1788795390.22 total 882.71 last 19.09) - open/close 14628 (time 1788795400.23 total 892.72 last 14.49) - open/close 14721 (time 1788795410.64 total 903.13 last 8.93) - open/close 14850 (time 1788795420.65 total 913.14 last 12.88) - open/close 14955 (time 1788795430.94 total 923.43 last 10.20) - open/close 15032 (time 1788795441.24 total 933.72 last 7.48) - open/close 15248 (time 1788795452.94 total 945.43 last 18.46) - open/close 15306 (time 1788795466.47 total 958.95 last 4.29) - open/close 15537 (time 1788795477.10 total 969.58 last 21.73) - open/close 15666 (time 1788795488.14 total 980.63 last 11.68) - open/close 15767 (time 1788795498.85 total 991.34 last 9.43) - open/close 15848 (time 1788795509.11 total 1001.60 last 7.89) - open/close 16018 (time 1788795520.16 total 1012.65 last 15.39) - open/close 16148 (time 1788795530.17 total 1022.66 last 12.99) - open/close 16302 (time 1788795540.18 total 1032.66 last 15.39) - open/close 16421 (time 1788795550.99 total 1043.48 last 11.00) - open/close 16558 (time 1788795561.29 total 1053.78 last 13.30) - open/close 16872 (time 1788795571.30 total 1063.79 last 31.38) - open/close 17510 (time 1788795581.31 total 1073.80 last 63.73) - open/close 18122 (time 1788795591.32 total 1083.81 last 61.13) - open/close 18724 (time 1788795601.33 total 1093.81 last 60.17) - open/close 19285 (time 1788795611.34 total 1103.83 last 56.03) - open/close 19963 (time 1788795621.34 total 1113.83 last 67.80) - open/close 20526 (time 1788795631.34 total 1123.83 last 56.30) - open/close 21182 (time 1788795641.35 total 1133.83 last 65.55) - open/close 21760 (time 1788795651.37 total 1143.85 last 57.68) - open/close 22438 (time 1788795661.39 total 1153.88 last 67.63) - open/close 23022 (time 1788795671.41 total 1163.89 last 58.32) - open/close 23643 (time 1788795681.41 total 1173.90 last 62.07) - open/close 24266 (time 1788795691.41 total 1183.90 last 62.29) - open/close 24892 (time 1788795701.42 total 1193.91 last 62.56) - open/close 25534 (time 1788795711.42 total 1203.91 last 64.20) - open/close 26107 (time 1788795721.44 total 1213.92 last 57.21) - open/close 26727 (time 1788795731.45 total 1223.94 last 61.89) - open/close 27331 (time 1788795741.46 total 1233.95 last 60.35) - open/close 27984 (time 1788795751.48 total 1243.96 last 65.22) - open/close 28615 (time 1788795761.48 total 1253.97 last 63.04) - open/close 29175 (time 1788795771.49 total 1263.98 last 55.95) - open/close 29601 (time 1788795781.50 total 1273.99 last 42.57) - open/close 29987 (time 1788795791.51 total 1283.99 last 38.58) - open/close 30608 (time 1788795801.52 total 1294.01 last 62.00) - open/close 31242 (time 1788795811.54 total 1304.03 last 63.30) - open/close 31882 (time 1788795821.55 total 1314.04 last 63.91) - open/close 32454 (time 1788795831.56 total 1324.05 last 57.16) - open/close 33051 (time 1788795841.56 total 1334.05 last 59.69) - open/close 33662 (time 1788795851.57 total 1344.06 last 61.05) - open/close 34257 (time 1788795861.58 total 1354.06 last 59.46) - open/close 34790 (time 1788795871.59 total 1364.07 last 53.25) - open/close 35187 (time 1788795881.60 total 1374.09 last 39.65) - open/close 35732 (time 1788795891.61 total 1384.10 last 54.45) - open/close 36289 (time 1788795901.61 total 1394.10 last 55.67) - open/close 36834 (time 1788795911.63 total 1404.12 last 54.41) - open/close 37475 (time 1788795921.63 total 1414.12 last 64.08) - open/close 38111 (time 1788795931.64 total 1424.13 last 63.54) - open/close 38746 (time 1788795941.65 total 1434.13 last 63.48) - open/close 39346 (time 1788795951.65 total 1444.14 last 59.96) - open/close 39906 (time 1788795961.66 total 1454.15 last 55.95) - open/close 40480 (time 1788795971.67 total 1464.15 last 57.37) - open/close 41028 (time 1788795981.67 total 1474.16 last 54.77) - open/close 41645 (time 1788795991.67 total 1484.16 last 61.68) - open/close 42255 (time 1788796001.69 total 1494.18 last 60.90) - open/close 42917 (time 1788796011.70 total 1504.19 last 66.12) - open/close 43505 (time 1788796021.72 total 1514.20 last 58.71) - open/close 44160 (time 1788796031.73 total 1524.21 last 65.43) - open/close 44740 (time 1788796041.74 total 1534.23 last 57.93) - open/close 45382 (time 1788796051.75 total 1544.24 last 64.14) - open/close 46021 (time 1788796061.76 total 1554.24 last 63.86) - open/close 46668 (time 1788796071.76 total 1564.24 last 64.69) - open/close 47349 (time 1788796081.76 total 1574.24 last 68.10) - open/close 48026 (time 1788796091.77 total 1584.25 last 67.64) - open/close 48559 (time 1788796101.78 total 1594.27 last 53.23) - open/close 49105 (time 1788796111.82 total 1604.31 last 54.36) - open/close 49639 (time 1788796121.85 total 1614.34 last 53.26) total: 50000 open/close in 1620.89 seconds: 30.85 ops/second - unlinked 0 (time 1788796134 ; total 0 ; last 0) - unlinked 10000 (time 1788796316 ; total 182 ; last 182) - unlinked 20000 (time 1788796483 ; total 349 ; last 167) - unlinked 30000 (time 1788796656 ; total 522 ; last 173) - unlinked 40000 (time 1788796861 ; total 727 ; last 205) total: 50000 unlinks in 903 seconds: 55.370987 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg412-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg412-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg412-server stop mds service on oleg412-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg412-server unloading modules via unload_modules_local on: 'oleg412-server' oleg412-server: oleg412-server.virtnet: executing unload_modules_local oleg412-server: modules unloaded. pdsh@oleg412-client: oleg412-server: ssh exited with exit code 2 pdsh@oleg412-client: oleg412-client: ssh exited with exit code 2 pdsh@oleg412-client: oleg412-server: ssh exited with exit code 2 pdsh@oleg412-client: oleg412-client: ssh exited with exit code 2