== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:25:05 (1789118705) start mds service on oleg215-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg215-server' oleg215-server: oleg215-server.virtnet: executing load_modules_local oleg215-server: Loading modules from /home/green/git/lustre-release/lustre oleg215-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg215-server: detected 4 online CPUs by sysfs oleg215-server: MODOPTS_LIBCFS= oleg215-server: Force libcfs to create 2 CPU partitions oleg215-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg215-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg215-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg215-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Started lustre-OST0000 oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg215-client.virtnet: -o user_xattr,flock 192.168.202.115@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 178 (time 1789118803.69 total 10.78 last 16.51) - open/close 269 (time 1789118813.94 total 21.03 last 8.88) - open/close 445 (time 1789118826.91 total 34.01 last 13.57) - open/close 606 (time 1789118837.82 total 44.91 last 14.76) - open/close 682 (time 1789118847.83 total 54.92 last 7.59) - open/close 901 (time 1789118857.90 total 64.99 last 21.75) - open/close 1137 (time 1789118867.91 total 75.00 last 23.57) - open/close 1359 (time 1789118877.92 total 85.01 last 22.17) - open/close 1557 (time 1789118888.44 total 95.53 last 18.83) - open/close 1787 (time 1789118898.82 total 105.92 last 22.14) - open/close 2049 (time 1789118908.84 total 115.94 last 26.15) - open/close 2262 (time 1789118919.62 total 126.71 last 19.77) - open/close 2477 (time 1789118934.42 total 141.51 last 14.53) - open/close 2654 (time 1789118944.65 total 151.74 last 17.30) - open/close 2825 (time 1789118954.66 total 161.75 last 17.08) - open/close 2984 (time 1789118964.66 total 171.75 last 15.90) - open/close 3154 (time 1789118975.35 total 182.44 last 15.91) - open/close 3322 (time 1789118985.37 total 192.46 last 16.76) - open/close 3580 (time 1789118995.39 total 202.48 last 25.75) - open/close 3759 (time 1789119005.43 total 212.52 last 17.83) - open/close 4006 (time 1789119015.92 total 223.02 last 23.53) - open/close 4143 (time 1789119026.68 total 233.77 last 12.74) - open/close 4246 (time 1789119037.06 total 244.15 last 9.92) - open/close 4340 (time 1789119048.83 total 255.92 last 7.99) - open/close 4479 (time 1789119060.48 total 267.57 last 11.93) - open/close 4624 (time 1789119072.82 total 279.92 last 11.75) - open/close 4781 (time 1789119083.78 total 290.87 last 14.33) - open/close 4912 (time 1789119093.80 total 300.89 last 13.07) - open/close 5070 (time 1789119103.80 total 310.90 last 15.79) - open/close 5269 (time 1789119114.32 total 321.41 last 18.93) - open/close 5406 (time 1789119125.69 total 332.79 last 12.04) - open/close 5667 (time 1789119136.63 total 343.73 last 23.86) - open/close 5792 (time 1789119146.87 total 353.97 last 12.21) - open/close 5841 (time 1789119159.54 total 366.63 last 3.87) - open/close 6058 (time 1789119169.54 total 376.63 last 21.70) - open/close 6188 (time 1789119179.55 total 386.64 last 12.99) - open/close 6454 (time 1789119189.55 total 396.64 last 26.59) - open/close 6643 (time 1789119199.59 total 406.68 last 18.83) - open/close 6824 (time 1789119209.61 total 416.71 last 18.05) - open/close 6916 (time 1789119220.08 total 427.17 last 8.79) - open/close 7086 (time 1789119230.11 total 437.20 last 16.95) - open/close 7310 (time 1789119240.56 total 447.65 last 21.42) - open/close 7534 (time 1789119250.56 total 457.66 last 22.39) - open/close 7719 (time 1789119261.00 total 468.09 last 17.74) - open/close 7949 (time 1789119271.74 total 478.83 last 21.41) - open/close 8038 (time 1789119282.29 total 489.38 last 8.44) - open/close 8168 (time 1789119293.67 total 500.77 last 11.42) - open/close 8286 (time 1789119303.68 total 510.77 last 11.79) - open/close 8457 (time 1789119316.29 total 523.39 last 13.55) - open/close 8672 (time 1789119327.31 total 534.41 last 19.51) - open/close 8833 (time 1789119337.84 total 544.93 last 15.30) - open/close 9023 (time 1789119348.03 total 555.12 last 18.64) - open/close 9222 (time 1789119358.50 total 565.60 last 19.00) - open/close 9363 (time 1789119368.64 total 575.74 last 13.90) - open/close 9537 (time 1789119378.66 total 585.76 last 17.37) - open/close 9818 (time 1789119388.67 total 595.76 last 28.09) - open/close 10000 (time 1789119396.17 total 603.27 last 24.25) - open/close 10135 (time 1789119406.93 total 614.02 last 12.55) - open/close 10209 (time 1789119417.72 total 624.81 last 6.86) - open/close 10457 (time 1789119428.23 total 635.32 last 23.59) - open/close 10690 (time 1789119438.23 total 645.33 last 23.30) - open/close 10933 (time 1789119449.07 total 656.17 last 22.42) - open/close 11109 (time 1789119459.11 total 666.21 last 17.53) - open/close 11306 (time 1789119469.44 total 676.53 last 19.08) - open/close 11514 (time 1789119480.45 total 687.55 last 18.88) - open/close 11743 (time 1789119490.46 total 697.55 last 22.88) - open/close 12010 (time 1789119503.47 total 710.57 last 20.52) - open/close 12218 (time 1789119514.29 total 721.39 last 19.23) - open/close 12452 (time 1789119524.66 total 731.76 last 22.56) - open/close 12712 (time 1789119537.30 total 744.39 last 20.58) - open/close 12942 (time 1789119547.31 total 754.40 last 22.98) - open/close 13151 (time 1789119557.32 total 764.42 last 20.87) - open/close 13317 (time 1789119567.42 total 774.51 last 16.44) - open/close 13569 (time 1789119578.29 total 785.38 last 23.18) - open/close 13773 (time 1789119588.30 total 795.39 last 20.39) - open/close 13985 (time 1789119598.33 total 805.42 last 21.14) - open/close 14194 (time 1789119608.62 total 815.72 last 20.29) - open/close 14395 (time 1789119618.94 total 826.03 last 19.49) - open/close 14650 (time 1789119628.95 total 836.05 last 25.46) - open/close 14831 (time 1789119639.63 total 846.73 last 16.95) - open/close 15091 (time 1789119649.64 total 856.73 last 25.99) - open/close 15282 (time 1789119660.16 total 867.26 last 18.14) - open/close 15431 (time 1789119670.55 total 877.64 last 14.35) - open/close 15651 (time 1789119680.69 total 887.78 last 21.70) - open/close 15875 (time 1789119691.65 total 898.75 last 20.42) - open/close 16015 (time 1789119704.78 total 911.87 last 10.67) - open/close 16128 (time 1789119714.80 total 921.90 last 11.27) - open/close 16361 (time 1789119725.12 total 932.21 last 22.60) - open/close 16545 (time 1789119735.14 total 942.23 last 18.36) - open/close 16748 (time 1789119745.84 total 952.93 last 18.97) - open/close 16935 (time 1789119755.86 total 962.95 last 18.67) - open/close 17141 (time 1789119765.86 total 972.95 last 20.59) - open/close 17366 (time 1789119776.49 total 983.58 last 21.17) - open/close 17590 (time 1789119786.71 total 993.80 last 21.92) - open/close 18323 (time 1789119796.71 total 1003.80 last 73.28) - open/close 19150 (time 1789119806.72 total 1013.81 last 82.63) - open/close 19919 (time 1789119816.73 total 1023.82 last 76.83) - open/close 20666 (time 1789119826.73 total 1033.82 last 74.69) - open/close 21416 (time 1789119836.73 total 1043.83 last 74.97) - open/close 22166 (time 1789119846.74 total 1053.83 last 74.93) - open/close 22900 (time 1789119856.75 total 1063.84 last 73.34) - open/close 23602 (time 1789119866.76 total 1073.85 last 70.12) - open/close 24373 (time 1789119876.76 total 1083.86 last 77.06) - open/close 25121 (time 1789119886.77 total 1093.86 last 74.78) - open/close 25892 (time 1789119896.78 total 1103.87 last 77.03) - open/close 26607 (time 1789119906.78 total 1113.87 last 71.50) - open/close 27230 (time 1789119916.79 total 1123.88 last 62.24) - open/close 27733 (time 1789119926.80 total 1133.89 last 50.25) - open/close 28283 (time 1789119936.81 total 1143.90 last 54.93) - open/close 28988 (time 1789119946.82 total 1153.91 last 70.44) - open/close 29628 (time 1789119956.83 total 1163.92 last 63.94) - open/close 30000 (time 1789119962.11 total 1169.20 last 70.50) - open/close 30701 (time 1789119972.11 total 1179.21 last 70.05) - open/close 31478 (time 1789119982.11 total 1189.21 last 77.69) - open/close 32066 (time 1789119992.14 total 1199.24 last 58.62) - open/close 32677 (time 1789120002.16 total 1209.25 last 61.02) - open/close 33296 (time 1789120012.17 total 1219.27 last 61.80) - open/close 33789 (time 1789120022.19 total 1229.28 last 49.23) - open/close 34367 (time 1789120032.19 total 1239.29 last 57.77) - open/close 35143 (time 1789120042.20 total 1249.29 last 77.54) - open/close 35929 (time 1789120052.20 total 1259.30 last 78.57) - open/close 36584 (time 1789120062.21 total 1269.30 last 65.46) - open/close 37259 (time 1789120072.22 total 1279.31 last 67.43) - open/close 37973 (time 1789120082.23 total 1289.32 last 71.35) - open/close 38712 (time 1789120092.23 total 1299.32 last 73.90) - open/close 39431 (time 1789120102.25 total 1309.35 last 71.72) - open/close 40000 (time 1789120111.33 total 1318.42 last 62.70) - open/close 40701 (time 1789120121.33 total 1328.43 last 70.05) - open/close 41414 (time 1789120131.35 total 1338.44 last 71.19) - open/close 42212 (time 1789120141.36 total 1348.45 last 79.74) - open/close 42918 (time 1789120151.36 total 1358.45 last 70.60) - open/close 43478 (time 1789120161.38 total 1368.47 last 55.91) - open/close 44070 (time 1789120171.38 total 1378.47 last 59.19) - open/close 44588 (time 1789120181.39 total 1388.48 last 51.74) - open/close 45126 (time 1789120191.39 total 1398.49 last 53.76) - open/close 45781 (time 1789120201.40 total 1408.49 last 65.47) - open/close 46441 (time 1789120211.41 total 1418.50 last 65.93) - open/close 47233 (time 1789120221.41 total 1428.50 last 79.18) - open/close 48062 (time 1789120231.42 total 1438.51 last 82.83) - open/close 48797 (time 1789120241.42 total 1448.51 last 73.50) - open/close 49477 (time 1789120251.43 total 1458.53 last 67.92) total: 50000 open/close in 1465.30 seconds: 34.12 ops/second - unlinked 0 (time 1789120263 ; total 0 ; last 0) - unlinked 10000 (time 1789120416 ; total 153 ; last 153) - unlinked 20000 (time 1789120567 ; total 304 ; last 151) - unlinked 30000 (time 1789120719 ; total 456 ; last 152) - unlinked 40000 (time 1789120862 ; total 599 ; last 143) total: 50000 unlinks in 739 seconds: 67.658997 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg215-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg215-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg215-server stop mds service on oleg215-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg215-server unloading modules via unload_modules_local on: 'oleg215-server' oleg215-server: oleg215-server.virtnet: executing unload_modules_local oleg215-server: modules unloaded. pdsh@oleg215-client: oleg215-server: ssh exited with exit code 2 pdsh@oleg215-client: oleg215-client: ssh exited with exit code 2 pdsh@oleg215-client: oleg215-client: ssh exited with exit code 2 pdsh@oleg215-client: oleg215-server: ssh exited with exit code 2