== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:44:22 (1778654662) start mds service on oleg204-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg204-server' oleg204-server: oleg204-server.virtnet: executing load_modules_local oleg204-server: Loading modules from /home/green/git/lustre-release/lustre oleg204-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg204-server: detected 4 online CPUs by sysfs oleg204-server: MODOPTS_LIBCFS= oleg204-server: Force libcfs to create 2 CPU partitions oleg204-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg204-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg204-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg204-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-OST0000 oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg204-client.virtnet: -o user_xattr,flock 192.168.202.104@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 246 (time 1778654740.93 total 10.85 last 22.68) - open/close 518 (time 1778654751.74 total 21.66 last 25.16) - open/close 779 (time 1778654762.65 total 32.57 last 23.92) - open/close 970 (time 1778654772.69 total 42.61 last 19.02) - open/close 1083 (time 1778654782.71 total 52.63 last 11.27) - open/close 1394 (time 1778654793.78 total 63.70 last 28.10) - open/close 1659 (time 1778654803.79 total 73.71 last 26.48) - open/close 1865 (time 1778654814.08 total 84.01 last 20.01) - open/close 2072 (time 1778654824.76 total 94.69 last 19.38) - open/close 2219 (time 1778654835.14 total 105.06 last 14.17) - open/close 2422 (time 1778654845.15 total 115.07 last 20.28) - open/close 2653 (time 1778654855.46 total 125.38 last 22.42) - open/close 2775 (time 1778654865.49 total 135.41 last 12.16) - open/close 3007 (time 1778654876.36 total 146.28 last 21.35) - open/close 3278 (time 1778654887.14 total 157.06 last 25.14) - open/close 3507 (time 1778654897.70 total 167.62 last 21.68) - open/close 3653 (time 1778654908.08 total 178.00 last 14.07) - open/close 3802 (time 1778654918.46 total 188.38 last 14.35) - open/close 3967 (time 1778654928.47 total 198.39 last 16.49) - open/close 4103 (time 1778654938.48 total 208.40 last 13.59) - open/close 4262 (time 1778654948.81 total 218.73 last 15.40) - open/close 4369 (time 1778654959.79 total 229.72 last 9.74) - open/close 4545 (time 1778654971.54 total 241.46 last 14.99) - open/close 4645 (time 1778654981.56 total 251.48 last 9.98) - open/close 4785 (time 1778654993.75 total 263.67 last 11.48) - open/close 4942 (time 1778655003.96 total 273.88 last 15.37) - open/close 5162 (time 1778655013.97 total 283.89 last 21.98) - open/close 5306 (time 1778655025.87 total 295.79 last 12.11) - open/close 5498 (time 1778655035.89 total 305.82 last 19.14) - open/close 5599 (time 1778655046.45 total 316.38 last 9.56) - open/close 5750 (time 1778655056.51 total 326.43 last 15.01) - open/close 5868 (time 1778655066.66 total 336.58 last 11.63) - open/close 5999 (time 1778655076.67 total 346.59 last 13.08) - open/close 6090 (time 1778655087.61 total 357.53 last 8.32) - open/close 6167 (time 1778655098.35 total 368.27 last 7.17) - open/close 6317 (time 1778655108.38 total 378.30 last 14.95) - open/close 6460 (time 1778655118.40 total 388.32 last 14.27) - open/close 6527 (time 1778655128.78 total 398.70 last 6.46) - open/close 6714 (time 1778655139.32 total 409.24 last 17.74) - open/close 6858 (time 1778655150.02 total 419.94 last 13.46) - open/close 7025 (time 1778655160.89 total 430.81 last 15.35) - open/close 7208 (time 1778655171.80 total 441.72 last 16.78) - open/close 7407 (time 1778655181.86 total 451.78 last 19.78) - open/close 7556 (time 1778655192.02 total 461.94 last 14.67) - open/close 7793 (time 1778655202.02 total 471.94 last 23.69) - open/close 7902 (time 1778655212.02 total 481.94 last 10.90) - open/close 8030 (time 1778655222.02 total 491.95 last 12.79) - open/close 8117 (time 1778655232.69 total 502.61 last 8.15) - open/close 8213 (time 1778655242.88 total 512.80 last 9.43) - open/close 8354 (time 1778655253.52 total 523.44 last 13.24) - open/close 8513 (time 1778655264.12 total 534.04 last 15.00) - open/close 8587 (time 1778655274.69 total 544.61 last 7.00) - open/close 8820 (time 1778655284.72 total 554.64 last 23.23) - open/close 8916 (time 1778655295.09 total 565.01 last 9.26) - open/close 9220 (time 1778655305.23 total 575.15 last 29.97) - open/close 9365 (time 1778655315.24 total 585.16 last 14.49) - open/close 9451 (time 1778655326.12 total 596.04 last 7.90) - open/close 9579 (time 1778655337.08 total 607.00 last 11.68) - open/close 9787 (time 1778655347.08 total 617.00 last 20.80) - open/close 9945 (time 1778655357.10 total 627.02 last 15.77) - open/close 10179 (time 1778655367.11 total 637.03 last 23.39) - open/close 10344 (time 1778655378.13 total 648.05 last 14.97) - open/close 10461 (time 1778655388.14 total 658.06 last 11.69) - open/close 10555 (time 1778655398.42 total 668.34 last 9.15) - open/close 10628 (time 1778655408.77 total 678.70 last 7.05) - open/close 10694 (time 1778655421.13 total 691.06 last 5.34) - open/close 10795 (time 1778655431.15 total 701.07 last 10.08) - open/close 10874 (time 1778655441.50 total 711.42 last 7.64) - open/close 11030 (time 1778655452.41 total 722.33 last 14.30) - open/close 11183 (time 1778655463.42 total 733.34 last 13.90) - open/close 11396 (time 1778655474.37 total 744.29 last 19.45) - open/close 11524 (time 1778655485.31 total 755.23 last 11.70) - open/close 11633 (time 1778655495.42 total 765.34 last 10.78) - open/close 11694 (time 1778655505.62 total 775.54 last 5.98) - open/close 11796 (time 1778655515.63 total 785.55 last 10.19) - open/close 12021 (time 1778655525.63 total 795.55 last 22.49) - open/close 12164 (time 1778655537.22 total 807.15 last 12.33) - open/close 12284 (time 1778655547.39 total 817.31 last 11.80) - open/close 12489 (time 1778655557.70 total 827.62 last 19.90) - open/close 12728 (time 1778655567.70 total 837.63 last 23.88) - open/close 12884 (time 1778655578.20 total 848.13 last 14.86) - open/close 13113 (time 1778655588.82 total 858.74 last 21.57) - open/close 13168 (time 1778655598.84 total 868.76 last 5.49) - open/close 13304 (time 1778655609.21 total 879.13 last 13.12) - open/close 13517 (time 1778655619.84 total 889.76 last 20.04) - open/close 13770 (time 1778655629.86 total 899.79 last 25.23) - open/close 13943 (time 1778655639.91 total 909.83 last 17.22) - open/close 14088 (time 1778655650.20 total 920.12 last 14.09) - open/close 14184 (time 1778655660.93 total 930.85 last 8.95) - open/close 14309 (time 1778655671.69 total 941.62 last 11.61) - open/close 14539 (time 1778655682.37 total 952.29 last 21.54) - open/close 14813 (time 1778655692.39 total 962.31 last 27.35) - open/close 14974 (time 1778655702.45 total 972.37 last 16.01) - open/close 15203 (time 1778655712.45 total 982.37 last 22.89) - open/close 15420 (time 1778655722.77 total 992.70 last 21.02) - open/close 15515 (time 1778655733.18 total 1003.10 last 9.13) - open/close 15707 (time 1778655743.18 total 1013.10 last 19.20) - open/close 15855 (time 1778655755.83 total 1025.75 last 11.70) - open/close 15973 (time 1778655765.96 total 1035.88 last 11.65) - open/close 16072 (time 1778655776.77 total 1046.69 last 9.16) - open/close 16219 (time 1778655787.01 total 1056.93 last 14.36) - open/close 16388 (time 1778655797.65 total 1067.57 last 15.88) - open/close 16499 (time 1778655808.45 total 1078.37 last 10.27) - open/close 16601 (time 1778655818.85 total 1088.77 last 9.81) - open/close 17148 (time 1778655828.85 total 1098.77 last 54.69) - open/close 18005 (time 1778655838.86 total 1108.78 last 85.64) - open/close 18705 (time 1778655848.86 total 1118.78 last 69.98) - open/close 19462 (time 1778655858.86 total 1128.78 last 75.69) - open/close 20000 (time 1778655864.78 total 1134.70 last 90.85) - open/close 20775 (time 1778655874.79 total 1144.71 last 77.41) - open/close 21556 (time 1778655884.81 total 1154.73 last 77.98) - open/close 22126 (time 1778655894.82 total 1164.74 last 56.93) - open/close 22759 (time 1778655904.83 total 1174.76 last 63.22) - open/close 23280 (time 1778655914.84 total 1184.76 last 52.06) - open/close 23840 (time 1778655924.85 total 1194.77 last 55.97) - open/close 24705 (time 1778655934.85 total 1204.77 last 86.44) - open/close 25666 (time 1778655944.86 total 1214.78 last 96.08) - open/close 26519 (time 1778655954.87 total 1224.79 last 85.21) - open/close 27100 (time 1778655964.87 total 1234.80 last 58.06) - open/close 27887 (time 1778655974.88 total 1244.80 last 78.64) - open/close 28719 (time 1778655984.89 total 1254.81 last 83.15) - open/close 29552 (time 1778655994.89 total 1264.81 last 83.27) - open/close 30000 (time 1778656000.96 total 1270.88 last 73.89) - open/close 30691 (time 1778656010.96 total 1280.88 last 69.08) - open/close 31557 (time 1778656020.97 total 1290.89 last 86.51) - open/close 32347 (time 1778656030.98 total 1300.90 last 78.93) - open/close 33101 (time 1778656040.98 total 1310.90 last 75.37) - open/close 33690 (time 1778656050.98 total 1320.90 last 58.88) - open/close 34231 (time 1778656061.01 total 1330.93 last 53.97) - open/close 34787 (time 1778656071.02 total 1340.94 last 55.55) - open/close 35468 (time 1778656081.02 total 1350.94 last 68.08) - open/close 36159 (time 1778656091.03 total 1360.96 last 69.00) - open/close 36987 (time 1778656101.04 total 1370.96 last 82.72) - open/close 37598 (time 1778656111.05 total 1380.97 last 61.09) - open/close 38191 (time 1778656121.06 total 1390.98 last 59.21) - open/close 38951 (time 1778656131.06 total 1400.99 last 75.97) - open/close 39464 (time 1778656141.10 total 1411.02 last 51.14) - open/close 40000 (time 1778656150.99 total 1420.92 last 54.15) - open/close 40586 (time 1778656161.00 total 1430.92 last 58.58) - open/close 41146 (time 1778656171.00 total 1440.92 last 55.99) - open/close 41827 (time 1778656181.01 total 1450.93 last 68.03) - open/close 42459 (time 1778656191.02 total 1460.94 last 63.14) - open/close 42983 (time 1778656201.03 total 1470.95 last 52.34) - open/close 43534 (time 1778656211.07 total 1480.99 last 54.91) - open/close 44232 (time 1778656221.08 total 1491.00 last 69.72) - open/close 44967 (time 1778656231.09 total 1501.01 last 73.41) - open/close 45709 (time 1778656241.09 total 1511.01 last 74.17) - open/close 46324 (time 1778656251.10 total 1521.02 last 61.45) - open/close 46875 (time 1778656261.11 total 1531.03 last 55.07) - open/close 47547 (time 1778656271.11 total 1541.03 last 67.17) - open/close 48223 (time 1778656281.12 total 1551.04 last 67.51) - open/close 48827 (time 1778656291.13 total 1561.05 last 60.34) - open/close 49405 (time 1778656301.15 total 1571.07 last 57.73) - open/close 49869 (time 1778656311.16 total 1581.08 last 46.35) total: 50000 open/close in 1583.30 seconds: 31.58 ops/second - unlinked 0 (time 1778656319 ; total 0 ; last 0) - unlinked 10000 (time 1778656464 ; total 145 ; last 145) - unlinked 20000 (time 1778656631 ; total 312 ; last 167) - unlinked 30000 (time 1778656785 ; total 466 ; last 154) - unlinked 40000 (time 1778656943 ; total 624 ; last 158) total: 50000 unlinks in 789 seconds: 63.371357 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg204-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg204-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg204-server stop mds service on oleg204-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg204-server unloading modules via unload_modules_local on: 'oleg204-server' oleg204-server: oleg204-server.virtnet: executing unload_modules_local oleg204-server: modules unloaded. pdsh@oleg204-client: oleg204-client: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-server: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-server: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-client: ssh exited with exit code 2