== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:00:22 (1788930022) start mds service on oleg234-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg234-server' oleg234-server: oleg234-server.virtnet: executing load_modules_local oleg234-server: Loading modules from /home/green/git/lustre-release/lustre oleg234-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg234-server: detected 4 online CPUs by sysfs oleg234-server: MODOPTS_LIBCFS= oleg234-server: Force libcfs to create 2 CPU partitions oleg234-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg234-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg234-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg234-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0000 oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 151 (time 1788930117.08 total 10.01 last 15.08) - open/close 308 (time 1788930127.09 total 20.02 last 15.69) - open/close 503 (time 1788930137.38 total 30.31 last 18.95) - open/close 572 (time 1788930148.33 total 41.26 last 6.30) - open/close 670 (time 1788930158.88 total 51.81 last 9.29) - open/close 851 (time 1788930168.89 total 61.82 last 18.08) - open/close 905 (time 1788930178.91 total 71.84 last 5.39) - open/close 1146 (time 1788930191.94 total 84.87 last 18.49) - open/close 1325 (time 1788930203.03 total 95.96 last 16.14) - open/close 1536 (time 1788930213.24 total 106.17 last 20.67) - open/close 1796 (time 1788930223.40 total 116.33 last 25.60) - open/close 1938 (time 1788930233.81 total 126.74 last 13.64) - open/close 2109 (time 1788930243.82 total 136.75 last 17.08) - open/close 2247 (time 1788930253.82 total 146.75 last 13.79) - open/close 2376 (time 1788930264.40 total 157.33 last 12.19) - open/close 2530 (time 1788930277.46 total 170.38 last 11.80) - open/close 2698 (time 1788930287.63 total 180.56 last 16.51) - open/close 2865 (time 1788930297.80 total 190.73 last 16.42) - open/close 2982 (time 1788930307.81 total 200.74 last 11.69) - open/close 3157 (time 1788930320.87 total 213.80 last 13.40) - open/close 3403 (time 1788930331.77 total 224.70 last 22.57) - open/close 3574 (time 1788930341.78 total 234.71 last 17.08) - open/close 3736 (time 1788930351.79 total 244.72 last 16.19) - open/close 3937 (time 1788930361.80 total 254.73 last 20.08) - open/close 3980 (time 1788930372.30 total 265.23 last 4.09) - open/close 4171 (time 1788930382.33 total 275.25 last 19.05) - open/close 4320 (time 1788930393.30 total 286.23 last 13.58) - open/close 4466 (time 1788930403.30 total 296.23 last 14.60) - open/close 4578 (time 1788930413.36 total 306.29 last 11.14) - open/close 4631 (time 1788930423.40 total 316.33 last 5.28) - open/close 4742 (time 1788930438.56 total 331.49 last 7.33) - open/close 4865 (time 1788930449.33 total 342.25 last 11.42) - open/close 4889 (time 1788930460.26 total 353.18 last 2.20) - open/close 4964 (time 1788930470.27 total 363.19 last 7.49) - open/close 5178 (time 1788930480.29 total 373.22 last 21.36) - open/close 5379 (time 1788930490.38 total 383.31 last 19.92) - open/close 5599 (time 1788930500.40 total 393.33 last 21.95) - open/close 5792 (time 1788930510.40 total 403.33 last 19.30) - open/close 6032 (time 1788930520.42 total 413.35 last 23.95) - open/close 6191 (time 1788930530.43 total 423.36 last 15.89) - open/close 6275 (time 1788930540.95 total 433.88 last 7.99) - open/close 6374 (time 1788930551.70 total 444.63 last 9.21) - open/close 6475 (time 1788930564.84 total 457.77 last 7.69) - open/close 6586 (time 1788930574.89 total 467.82 last 11.04) - open/close 6715 (time 1788930585.43 total 478.36 last 12.24) - open/close 6839 (time 1788930595.60 total 488.53 last 12.19) - open/close 7028 (time 1788930605.61 total 498.53 last 18.89) - open/close 7230 (time 1788930615.61 total 508.54 last 20.18) - open/close 7374 (time 1788930625.64 total 518.57 last 14.36) - open/close 7468 (time 1788930636.45 total 529.38 last 8.70) - open/close 7535 (time 1788930647.58 total 540.51 last 6.02) - open/close 7683 (time 1788930657.66 total 550.58 last 14.69) - open/close 7835 (time 1788930668.11 total 561.04 last 14.54) - open/close 7921 (time 1788930678.13 total 571.06 last 8.59) - open/close 8004 (time 1788930688.23 total 581.16 last 8.22) - open/close 8111 (time 1788930699.10 total 592.03 last 9.84) - open/close 8257 (time 1788930710.76 total 603.69 last 12.51) - open/close 8406 (time 1788930721.19 total 614.12 last 14.29) - open/close 8550 (time 1788930732.14 total 625.06 last 13.16) - open/close 8687 (time 1788930743.08 total 636.01 last 12.51) - open/close 8833 (time 1788930753.96 total 646.89 last 13.42) - open/close 8996 (time 1788930764.75 total 657.68 last 15.11) - open/close 9076 (time 1788930775.12 total 668.05 last 7.71) - open/close 9260 (time 1788930785.37 total 678.29 last 17.96) - open/close 9526 (time 1788930795.37 total 688.30 last 26.58) - open/close 9704 (time 1788930805.38 total 698.31 last 17.79) - open/close 9889 (time 1788930815.38 total 708.31 last 18.49) - open/close 9966 (time 1788930825.62 total 718.55 last 7.52) - open/close 10096 (time 1788930836.19 total 729.12 last 12.29) - open/close 10267 (time 1788930846.21 total 739.14 last 17.07) - open/close 10393 (time 1788930856.21 total 749.14 last 12.60) - open/close 10554 (time 1788930866.97 total 759.90 last 14.97) - open/close 10663 (time 1788930876.98 total 769.91 last 10.89) - open/close 10769 (time 1788930888.42 total 781.35 last 9.27) - open/close 10823 (time 1788930898.44 total 791.37 last 5.39) - open/close 10940 (time 1788930908.50 total 801.43 last 11.63) - open/close 11106 (time 1788930918.53 total 811.46 last 16.56) - open/close 11200 (time 1788930928.55 total 821.48 last 9.38) - open/close 11286 (time 1788930940.11 total 833.04 last 7.44) - open/close 11394 (time 1788930950.11 total 843.04 last 10.80) - open/close 11640 (time 1788930960.79 total 853.72 last 23.04) - open/close 11786 (time 1788930970.81 total 863.74 last 14.57) - open/close 11845 (time 1788930980.81 total 873.74 last 5.90) - open/close 12072 (time 1788930990.85 total 883.78 last 22.61) - open/close 12199 (time 1788931001.36 total 894.29 last 12.08) - open/close 12254 (time 1788931011.38 total 904.31 last 5.49) - open/close 12419 (time 1788931021.81 total 914.74 last 15.83) - open/close 12523 (time 1788931032.30 total 925.23 last 9.91) - open/close 12635 (time 1788931042.64 total 935.57 last 10.84) - open/close 12782 (time 1788931052.64 total 945.57 last 14.69) - open/close 12869 (time 1788931063.63 total 956.56 last 7.92) - open/close 12983 (time 1788931074.20 total 967.13 last 10.79) - open/close 13065 (time 1788931085.08 total 978.01 last 7.54) - open/close 13122 (time 1788931095.09 total 988.02 last 5.70) - open/close 13215 (time 1788931105.97 total 998.90 last 8.55) - open/close 13260 (time 1788931116.59 total 1009.52 last 4.24) - open/close 13386 (time 1788931127.13 total 1020.06 last 11.96) - open/close 13428 (time 1788931138.01 total 1030.94 last 3.86) - open/close 13533 (time 1788931148.02 total 1040.95 last 10.48) - open/close 13660 (time 1788931158.75 total 1051.68 last 11.84) - open/close 13824 (time 1788931169.83 total 1062.76 last 14.81) - open/close 13867 (time 1788931180.49 total 1073.42 last 4.03) - open/close 14009 (time 1788931191.34 total 1084.27 last 13.09) - open/close 14184 (time 1788931201.34 total 1094.27 last 17.50) - open/close 14344 (time 1788931211.35 total 1104.28 last 15.98) - open/close 14477 (time 1788931221.45 total 1114.38 last 13.17) - open/close 14590 (time 1788931231.47 total 1124.40 last 11.28) - open/close 14695 (time 1788931241.95 total 1134.88 last 10.02) - open/close 14829 (time 1788931252.13 total 1145.06 last 13.17) - open/close 15281 (time 1788931262.15 total 1155.08 last 45.13) - open/close 15867 (time 1788931272.16 total 1165.09 last 58.54) - open/close 16433 (time 1788931282.16 total 1175.09 last 56.56) - open/close 16943 (time 1788931292.20 total 1185.13 last 50.83) - open/close 17480 (time 1788931302.22 total 1195.14 last 53.61) - open/close 18024 (time 1788931312.24 total 1205.17 last 54.24) - open/close 18459 (time 1788931322.27 total 1215.20 last 43.39) - open/close 18900 (time 1788931332.28 total 1225.21 last 44.03) - open/close 19339 (time 1788931342.29 total 1235.22 last 43.87) - open/close 19730 (time 1788931352.30 total 1245.23 last 39.06) - open/close 20000 (time 1788931357.91 total 1250.84 last 48.11) - open/close 20473 (time 1788931367.92 total 1260.85 last 47.28) - open/close 21032 (time 1788931377.93 total 1270.86 last 55.85) - open/close 21510 (time 1788931387.93 total 1280.86 last 47.78) - open/close 22023 (time 1788931397.97 total 1290.90 last 51.11) - open/close 22535 (time 1788931407.98 total 1300.90 last 51.17) - open/close 23058 (time 1788931417.98 total 1310.91 last 52.26) - open/close 23584 (time 1788931427.98 total 1320.91 last 52.59) - open/close 24048 (time 1788931437.99 total 1330.92 last 46.38) - open/close 24539 (time 1788931448.00 total 1340.93 last 49.04) - open/close 25119 (time 1788931458.00 total 1350.93 last 57.99) - open/close 25712 (time 1788931468.01 total 1360.94 last 59.25) - open/close 26313 (time 1788931478.01 total 1370.94 last 60.08) - open/close 26927 (time 1788931488.02 total 1380.95 last 61.36) - open/close 27562 (time 1788931498.02 total 1390.95 last 63.49) - open/close 28162 (time 1788931508.02 total 1400.95 last 59.97) - open/close 28826 (time 1788931518.05 total 1410.98 last 66.23) - open/close 29377 (time 1788931528.06 total 1420.99 last 55.04) - open/close 29992 (time 1788931538.07 total 1431.00 last 61.46) - open/close 30572 (time 1788931548.07 total 1441.00 last 57.96) - open/close 31228 (time 1788931558.08 total 1451.01 last 65.57) - open/close 31879 (time 1788931568.09 total 1461.02 last 65.02) - open/close 32464 (time 1788931578.11 total 1471.03 last 58.41) - open/close 33214 (time 1788931588.11 total 1481.04 last 74.94) - open/close 33886 (time 1788931598.12 total 1491.05 last 67.18) - open/close 34491 (time 1788931608.12 total 1501.05 last 60.50) - open/close 35111 (time 1788931618.13 total 1511.06 last 61.94) - open/close 35720 (time 1788931628.13 total 1521.06 last 60.86) - open/close 36343 (time 1788931638.15 total 1531.08 last 62.21) - open/close 36972 (time 1788931648.16 total 1541.09 last 62.85) - open/close 37668 (time 1788931658.16 total 1551.09 last 69.58) - open/close 38261 (time 1788931668.16 total 1561.09 last 59.28) - open/close 38908 (time 1788931678.17 total 1571.10 last 64.65) - open/close 39547 (time 1788931688.18 total 1581.11 last 63.81) - open/close 40000 (time 1788931695.05 total 1587.98 last 66.01) - open/close 40590 (time 1788931705.06 total 1597.99 last 58.94) - open/close 41210 (time 1788931715.08 total 1608.01 last 61.88) - open/close 41797 (time 1788931725.08 total 1618.01 last 58.66) - open/close 42467 (time 1788931735.09 total 1628.01 last 66.99) - open/close 43109 (time 1788931745.10 total 1638.03 last 64.11) - open/close 43777 (time 1788931755.10 total 1648.03 last 66.80) - open/close 44400 (time 1788931765.12 total 1658.04 last 62.21) - open/close 44989 (time 1788931775.13 total 1668.05 last 58.84) - open/close 45593 (time 1788931785.13 total 1678.06 last 60.37) - open/close 46180 (time 1788931795.15 total 1688.08 last 58.57) - open/close 46841 (time 1788931805.17 total 1698.09 last 66.02) - open/close 47526 (time 1788931815.18 total 1708.11 last 68.39) - open/close 48142 (time 1788931825.18 total 1718.11 last 61.58) - open/close 48805 (time 1788931835.19 total 1728.12 last 66.28) - open/close 49445 (time 1788931845.21 total 1738.14 last 63.88) total: 50000 open/close in 1747.11 seconds: 28.62 ops/second - unlinked 0 (time 1788931858 ; total 0 ; last 0) - unlinked 10000 (time 1788932012 ; total 154 ; last 154) - unlinked 20000 (time 1788932160 ; total 302 ; last 148) - unlinked 30000 (time 1788932318 ; total 460 ; last 158) - unlinked 40000 (time 1788932489 ; total 631 ; last 171) total: 50000 unlinks in 788 seconds: 63.451778 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg234-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg234-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg234-server stop mds service on oleg234-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg234-server unloading modules via unload_modules_local on: 'oleg234-server' oleg234-server: oleg234-server.virtnet: executing unload_modules_local oleg234-server: modules unloaded. pdsh@oleg234-client: oleg234-client: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-server: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-client: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-server: ssh exited with exit code 2