== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 08:44:01 (1787834641) start mds service on oleg117-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg117-server' oleg117-server: oleg117-server.virtnet: executing load_modules_local oleg117-server: Loading modules from /home/green/git/lustre-release/lustre oleg117-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg117-server: detected 4 online CPUs by sysfs oleg117-server: MODOPTS_LIBCFS= oleg117-server: Force libcfs to create 2 CPU partitions oleg117-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg117-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg117-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg117-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg117-client.virtnet: -o user_xattr,flock 192.168.201.117@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 239 (time 1787834726.75 total 10.86 last 22.01) - open/close 509 (time 1787834737.70 total 21.81 last 24.66) - open/close 751 (time 1787834747.70 total 31.81 last 24.20) - open/close 860 (time 1787834757.72 total 41.83 last 10.88) - open/close 1054 (time 1787834768.23 total 52.34 last 18.44) - open/close 1317 (time 1787834778.25 total 62.36 last 26.27) - open/close 1505 (time 1787834788.26 total 72.37 last 18.78) - open/close 1795 (time 1787834798.26 total 82.37 last 28.99) - open/close 2012 (time 1787834808.29 total 92.40 last 21.63) - open/close 2254 (time 1787834818.30 total 102.41 last 24.20) - open/close 2509 (time 1787834828.60 total 112.71 last 24.76) - open/close 2726 (time 1787834839.69 total 123.80 last 19.57) - open/close 2843 (time 1787834852.38 total 136.48 last 9.22) - open/close 3054 (time 1787834864.88 total 148.99 last 16.88) - open/close 3201 (time 1787834874.88 total 158.99 last 14.70) - open/close 3331 (time 1787834884.92 total 169.03 last 12.95) - open/close 3407 (time 1787834897.19 total 181.30 last 6.19) - open/close 3563 (time 1787834909.22 total 193.33 last 12.97) - open/close 3762 (time 1787834919.24 total 203.35 last 19.86) - open/close 3987 (time 1787834929.64 total 213.75 last 21.63) - open/close 4259 (time 1787834939.77 total 223.88 last 26.85) - open/close 4487 (time 1787834950.05 total 234.16 last 22.18) - open/close 4718 (time 1787834960.34 total 244.45 last 22.45) - open/close 4907 (time 1787834971.10 total 255.21 last 17.57) - open/close 5040 (time 1787834981.14 total 265.25 last 13.25) - open/close 5183 (time 1787834991.16 total 275.27 last 14.28) - open/close 5410 (time 1787835001.17 total 285.28 last 22.68) - open/close 5642 (time 1787835011.24 total 295.35 last 23.03) - open/close 5854 (time 1787835021.37 total 305.48 last 20.92) - open/close 6102 (time 1787835032.04 total 316.15 last 23.26) - open/close 6364 (time 1787835042.72 total 326.83 last 24.51) - open/close 6631 (time 1787835053.34 total 337.45 last 25.15) - open/close 6894 (time 1787835063.90 total 348.01 last 24.90) - open/close 7121 (time 1787835073.92 total 358.03 last 22.66) - open/close 7235 (time 1787835086.00 total 370.11 last 9.43) - open/close 7489 (time 1787835096.87 total 380.98 last 23.37) - open/close 7692 (time 1787835107.23 total 391.34 last 19.60) - open/close 7903 (time 1787835117.23 total 401.34 last 21.09) - open/close 8123 (time 1787835128.09 total 412.20 last 20.27) - open/close 8360 (time 1787835138.09 total 422.20 last 23.69) - open/close 8600 (time 1787835148.10 total 432.21 last 23.99) - open/close 8841 (time 1787835158.17 total 442.28 last 23.93) - open/close 9086 (time 1787835168.22 total 452.32 last 24.39) - open/close 9309 (time 1787835178.39 total 462.50 last 21.92) - open/close 9599 (time 1787835188.40 total 472.51 last 28.96) - open/close 9875 (time 1787835199.35 total 483.46 last 25.21) - open/close 10000 (time 1787835204.47 total 488.58 last 24.40) - open/close 10224 (time 1787835215.46 total 499.57 last 20.39) - open/close 10464 (time 1787835225.59 total 509.70 last 23.69) - open/close 10689 (time 1787835236.51 total 520.61 last 20.61) - open/close 10938 (time 1787835246.51 total 530.62 last 24.88) - open/close 11073 (time 1787835257.39 total 541.50 last 12.41) - open/close 11163 (time 1787835267.51 total 551.62 last 8.89) - open/close 11366 (time 1787835277.51 total 561.62 last 20.29) - open/close 11555 (time 1787835288.23 total 572.34 last 17.63) - open/close 11758 (time 1787835298.84 total 582.95 last 19.13) - open/close 11947 (time 1787835309.33 total 593.44 last 18.02) - open/close 12216 (time 1787835320.22 total 604.33 last 24.71) - open/close 12484 (time 1787835331.17 total 615.28 last 24.49) - open/close 12718 (time 1787835342.11 total 626.22 last 21.39) - open/close 12951 (time 1787835352.12 total 636.23 last 23.27) - open/close 13160 (time 1787835362.87 total 646.98 last 19.44) - open/close 13363 (time 1787835375.34 total 659.45 last 16.27) - open/close 13708 (time 1787835386.34 total 670.45 last 31.38) - open/close 13939 (time 1787835397.14 total 681.25 last 21.37) - open/close 14151 (time 1787835407.96 total 692.07 last 19.60) - open/close 14392 (time 1787835418.17 total 702.28 last 23.60) - open/close 14642 (time 1787835428.83 total 712.94 last 23.45) - open/close 14867 (time 1787835438.84 total 722.95 last 22.48) - open/close 15087 (time 1787835448.93 total 733.04 last 21.81) - open/close 15335 (time 1787835458.93 total 743.04 last 24.80) - open/close 15612 (time 1787835469.24 total 753.35 last 26.86) - open/close 15836 (time 1787835479.25 total 763.36 last 22.39) - open/close 16050 (time 1787835489.26 total 773.37 last 21.38) - open/close 16303 (time 1787835499.67 total 783.78 last 24.29) - open/close 16504 (time 1787835509.85 total 793.96 last 19.75) - open/close 16659 (time 1787835520.17 total 804.28 last 15.03) - open/close 16883 (time 1787835531.17 total 815.28 last 20.35) - open/close 17063 (time 1787835542.70 total 826.81 last 15.61) - open/close 17244 (time 1787835553.03 total 837.14 last 17.53) - open/close 17512 (time 1787835563.04 total 847.15 last 26.75) - open/close 17764 (time 1787835573.54 total 857.65 last 24.00) - open/close 17893 (time 1787835583.56 total 867.67 last 12.88) - open/close 18151 (time 1787835593.56 total 877.67 last 25.80) - open/close 18395 (time 1787835604.27 total 888.38 last 22.78) - open/close 18637 (time 1787835614.28 total 898.39 last 24.18) - open/close 19452 (time 1787835624.28 total 908.39 last 81.49) - open/close 20000 (time 1787835632.04 total 916.15 last 70.58) - open/close 20752 (time 1787835642.05 total 926.16 last 75.14) - open/close 21415 (time 1787835652.07 total 936.18 last 66.17) - open/close 22164 (time 1787835662.07 total 946.18 last 74.89) - open/close 22944 (time 1787835672.08 total 956.19 last 77.95) - open/close 23859 (time 1787835682.08 total 966.19 last 91.48) - open/close 24746 (time 1787835692.09 total 976.20 last 88.66) - open/close 25357 (time 1787835702.09 total 986.20 last 61.05) - open/close 25911 (time 1787835712.11 total 996.22 last 55.32) - open/close 26571 (time 1787835722.12 total 1006.23 last 65.95) - open/close 27155 (time 1787835732.15 total 1016.26 last 58.23) - open/close 27751 (time 1787835742.18 total 1026.29 last 59.41) - open/close 28470 (time 1787835752.18 total 1036.29 last 71.88) - open/close 29336 (time 1787835762.18 total 1046.29 last 86.59) - open/close 30000 (time 1787835770.59 total 1054.70 last 78.92) - open/close 30814 (time 1787835780.60 total 1064.71 last 81.35) - open/close 31628 (time 1787835790.61 total 1074.72 last 81.34) - open/close 32500 (time 1787835800.61 total 1084.72 last 87.17) - open/close 33338 (time 1787835810.61 total 1094.72 last 83.77) - open/close 34033 (time 1787835820.63 total 1104.74 last 69.40) - open/close 34695 (time 1787835830.63 total 1114.74 last 66.18) - open/close 35411 (time 1787835840.64 total 1124.75 last 71.58) - open/close 36198 (time 1787835850.64 total 1134.75 last 78.63) - open/close 36967 (time 1787835860.65 total 1144.76 last 76.84) - open/close 37786 (time 1787835870.65 total 1154.76 last 81.88) - open/close 38598 (time 1787835880.67 total 1164.78 last 81.05) - open/close 39220 (time 1787835890.79 total 1174.90 last 61.50) - open/close 39967 (time 1787835900.81 total 1184.92 last 74.49) - open/close 40624 (time 1787835910.82 total 1194.93 last 65.67) - open/close 41319 (time 1787835920.82 total 1204.93 last 69.48) - open/close 42057 (time 1787835930.84 total 1214.95 last 73.64) - open/close 42703 (time 1787835940.84 total 1224.95 last 64.59) - open/close 43368 (time 1787835950.85 total 1234.95 last 66.49) - open/close 44154 (time 1787835960.85 total 1244.96 last 78.58) - open/close 44861 (time 1787835970.85 total 1254.96 last 70.66) - open/close 45514 (time 1787835980.86 total 1264.97 last 65.23) - open/close 46117 (time 1787835990.87 total 1274.98 last 60.27) - open/close 46718 (time 1787836000.88 total 1284.99 last 60.05) - open/close 47428 (time 1787836010.88 total 1294.99 last 70.96) - open/close 48205 (time 1787836020.88 total 1304.99 last 77.69) - open/close 48836 (time 1787836030.89 total 1315.00 last 63.07) - open/close 49564 (time 1787836040.90 total 1325.01 last 72.68) total: 50000 open/close in 1332.29 seconds: 37.53 ops/second - unlinked 0 (time 1787836055 ; total 0 ; last 0) - unlinked 10000 (time 1787836193 ; total 138 ; last 138) - unlinked 20000 (time 1787836328 ; total 273 ; last 135) - unlinked 30000 (time 1787836449 ; total 394 ; last 121) - unlinked 40000 (time 1787836589 ; total 534 ; last 140) total: 50000 unlinks in 669 seconds: 74.738419 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg117-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg117-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg117-server stop mds service on oleg117-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg117-server unloading modules via unload_modules_local on: 'oleg117-server' oleg117-server: oleg117-server.virtnet: executing unload_modules_local oleg117-server: modules unloaded. pdsh@oleg117-client: oleg117-client: ssh exited with exit code 2 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 2 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 2 pdsh@oleg117-client: oleg117-client: ssh exited with exit code 2