== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 03:11:09 (1785222669) start mds service on oleg121-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg121-server' oleg121-server: oleg121-server.virtnet: executing load_modules_local oleg121-server: Loading modules from /home/green/git/lustre-release/lustre oleg121-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg121-server: detected 4 online CPUs by sysfs oleg121-server: MODOPTS_LIBCFS= oleg121-server: Force libcfs to create 2 CPU partitions oleg121-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg121-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg121-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg121-server: oleg121-server.virtnet: executing set_default_debug -1 all pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg121-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg121-server: oleg121-server.virtnet: executing set_default_debug -1 all pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 Started lustre-OST0000 oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg121-client.virtnet: -o user_xattr,flock 192.168.201.121@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 193 (time 1785222736.58 total 10.01 last 19.27) - open/close 399 (time 1785222746.58 total 20.01 last 20.60) - open/close 610 (time 1785222756.65 total 30.09 last 20.94) - open/close 904 (time 1785222767.10 total 40.53 last 28.15) - open/close 1127 (time 1785222777.77 total 51.21 last 20.89) - open/close 1333 (time 1785222787.95 total 61.39 last 20.24) - open/close 1505 (time 1785222798.14 total 71.58 last 16.88) - open/close 1689 (time 1785222808.94 total 82.37 last 17.04) - open/close 1880 (time 1785222819.56 total 92.99 last 17.98) - open/close 2042 (time 1785222829.57 total 103.01 last 16.18) - open/close 2191 (time 1785222839.73 total 113.17 last 14.66) - open/close 2366 (time 1785222850.15 total 123.59 last 16.80) - open/close 2504 (time 1785222860.39 total 133.83 last 13.47) - open/close 2680 (time 1785222871.08 total 144.52 last 16.46) - open/close 2818 (time 1785222881.25 total 154.69 last 13.57) - open/close 2982 (time 1785222891.69 total 165.12 last 15.72) - open/close 3104 (time 1785222902.70 total 176.14 last 11.08) - open/close 3216 (time 1785222913.48 total 186.91 last 10.39) - open/close 3419 (time 1785222924.39 total 197.83 last 18.60) - open/close 3556 (time 1785222934.51 total 207.94 last 13.55) - open/close 3703 (time 1785222945.58 total 219.01 last 13.28) - open/close 3914 (time 1785222956.26 total 229.69 last 19.75) - open/close 4062 (time 1785222967.08 total 240.52 last 13.68) - open/close 4301 (time 1785222977.26 total 250.70 last 23.48) - open/close 4485 (time 1785222988.08 total 261.51 last 17.02) - open/close 4709 (time 1785222998.45 total 271.89 last 21.59) - open/close 4926 (time 1785223008.82 total 282.26 last 20.92) - open/close 5140 (time 1785223018.87 total 292.31 last 21.29) - open/close 5236 (time 1785223028.92 total 302.36 last 9.56) - open/close 5351 (time 1785223038.93 total 312.36 last 11.49) - open/close 5470 (time 1785223049.26 total 322.69 last 11.52) - open/close 5602 (time 1785223059.84 total 333.27 last 12.47) - open/close 5761 (time 1785223070.53 total 343.96 last 14.88) - open/close 5989 (time 1785223081.52 total 354.95 last 20.75) - open/close 6154 (time 1785223092.37 total 365.80 last 15.20) - open/close 6224 (time 1785223102.48 total 375.92 last 6.92) - open/close 6433 (time 1785223112.52 total 385.95 last 20.82) - open/close 6574 (time 1785223123.06 total 396.49 last 13.38) - open/close 6801 (time 1785223134.08 total 407.52 last 20.59) - open/close 7011 (time 1785223144.43 total 417.86 last 20.30) - open/close 7240 (time 1785223154.44 total 427.88 last 22.87) - open/close 7444 (time 1785223164.98 total 438.41 last 19.36) - open/close 7635 (time 1785223174.98 total 448.41 last 19.10) - open/close 7870 (time 1785223185.99 total 459.43 last 21.33) - open/close 8134 (time 1785223196.52 total 469.96 last 25.07) - open/close 8376 (time 1785223207.02 total 480.46 last 23.05) - open/close 8582 (time 1785223217.08 total 490.52 last 20.47) - open/close 8817 (time 1785223227.11 total 500.55 last 23.43) - open/close 9014 (time 1785223237.23 total 510.67 last 19.46) - open/close 9140 (time 1785223247.26 total 520.69 last 12.57) - open/close 9318 (time 1785223258.03 total 531.47 last 16.52) - open/close 9451 (time 1785223269.11 total 542.55 last 12.00) - open/close 9679 (time 1785223279.87 total 553.31 last 21.19) - open/close 9865 (time 1785223290.16 total 563.60 last 18.08) - open/close 10000 (time 1785223294.81 total 568.25 last 29.03) - open/close 10194 (time 1785223304.94 total 578.38 last 19.15) - open/close 10417 (time 1785223315.63 total 589.07 last 20.87) - open/close 10646 (time 1785223325.75 total 599.19 last 22.63) - open/close 10829 (time 1785223336.66 total 610.09 last 16.78) - open/close 11050 (time 1785223347.07 total 620.51 last 21.21) - open/close 11299 (time 1785223357.09 total 630.52 last 24.87) - open/close 11482 (time 1785223367.44 total 640.87 last 17.68) - open/close 11672 (time 1785223377.65 total 651.09 last 18.61) - open/close 11840 (time 1785223388.69 total 662.13 last 15.22) - open/close 12034 (time 1785223398.70 total 672.13 last 19.39) - open/close 12169 (time 1785223408.88 total 682.32 last 13.26) - open/close 12400 (time 1785223419.27 total 692.70 last 22.24) - open/close 12688 (time 1785223429.30 total 702.73 last 28.72) - open/close 12905 (time 1785223440.01 total 713.44 last 20.26) - open/close 13142 (time 1785223450.03 total 723.46 last 23.65) - open/close 13321 (time 1785223460.53 total 733.96 last 17.04) - open/close 13601 (time 1785223470.54 total 743.98 last 27.96) - open/close 13834 (time 1785223480.55 total 753.98 last 23.28) - open/close 14010 (time 1785223490.90 total 764.34 last 17.00) - open/close 14252 (time 1785223503.53 total 776.97 last 19.16) - open/close 14396 (time 1785223514.25 total 787.68 last 13.44) - open/close 14510 (time 1785223524.54 total 797.98 last 11.08) - open/close 14612 (time 1785223534.56 total 807.99 last 10.18) - open/close 14753 (time 1785223545.07 total 818.51 last 13.41) - open/close 15043 (time 1785223555.08 total 828.52 last 28.98) - open/close 15159 (time 1785223565.10 total 838.53 last 11.58) - open/close 15517 (time 1785223575.13 total 848.57 last 35.69) - open/close 16223 (time 1785223585.13 total 858.57 last 70.59) - open/close 16915 (time 1785223595.14 total 868.57 last 69.17) - open/close 17538 (time 1785223605.15 total 878.58 last 62.24) - open/close 18302 (time 1785223615.15 total 888.59 last 76.39) - open/close 18928 (time 1785223625.15 total 898.59 last 62.58) - open/close 19604 (time 1785223635.18 total 908.61 last 67.42) - open/close 20000 (time 1785223640.73 total 914.17 last 71.33) - open/close 20679 (time 1785223650.76 total 924.20 last 67.69) - open/close 21221 (time 1785223660.77 total 934.20 last 54.17) - open/close 21808 (time 1785223670.77 total 944.20 last 58.70) - open/close 22415 (time 1785223680.77 total 954.20 last 60.69) - open/close 23044 (time 1785223690.77 total 964.21 last 62.88) - open/close 23630 (time 1785223700.78 total 974.22 last 58.56) - open/close 24272 (time 1785223710.79 total 984.22 last 64.15) - open/close 24948 (time 1785223720.80 total 994.23 last 67.53) - open/close 25607 (time 1785223730.81 total 1004.24 last 65.84) - open/close 26247 (time 1785223740.81 total 1014.25 last 63.98) - open/close 26921 (time 1785223750.82 total 1024.26 last 67.33) - open/close 27367 (time 1785223760.83 total 1034.26 last 44.58) - open/close 27995 (time 1785223770.84 total 1044.28 last 62.69) - open/close 28643 (time 1785223780.85 total 1054.28 last 64.77) - open/close 29280 (time 1785223790.86 total 1064.29 last 63.64) - open/close 29897 (time 1785223800.86 total 1074.30 last 61.67) - open/close 30614 (time 1785223810.87 total 1084.31 last 71.62) - open/close 31235 (time 1785223820.88 total 1094.31 last 62.09) - open/close 31888 (time 1785223830.88 total 1104.32 last 65.24) - open/close 32561 (time 1785223840.89 total 1114.32 last 67.27) - open/close 33234 (time 1785223850.90 total 1124.33 last 67.26) - open/close 33917 (time 1785223860.90 total 1134.34 last 68.26) - open/close 34666 (time 1785223870.93 total 1144.36 last 74.70) - open/close 35374 (time 1785223880.94 total 1154.37 last 70.72) - open/close 36007 (time 1785223890.96 total 1164.39 last 63.20) - open/close 36632 (time 1785223900.96 total 1174.40 last 62.44) - open/close 37222 (time 1785223910.97 total 1184.40 last 59.00) - open/close 37892 (time 1785223920.99 total 1194.42 last 66.86) - open/close 38538 (time 1785223930.99 total 1204.43 last 64.57) - open/close 39246 (time 1785223941.00 total 1214.44 last 70.72) - open/close 39954 (time 1785223951.01 total 1224.44 last 70.76) - open/close 40705 (time 1785223961.02 total 1234.46 last 74.98) - open/close 41363 (time 1785223971.04 total 1244.48 last 65.69) - open/close 42034 (time 1785223981.05 total 1254.49 last 67.03) - open/close 42690 (time 1785223991.08 total 1264.51 last 65.42) - open/close 43289 (time 1785224001.09 total 1274.53 last 59.82) - open/close 43968 (time 1785224011.10 total 1284.53 last 67.88) - open/close 44643 (time 1785224021.10 total 1294.54 last 67.43) - open/close 45262 (time 1785224031.11 total 1304.54 last 61.90) - open/close 45940 (time 1785224041.11 total 1314.54 last 67.77) - open/close 46580 (time 1785224051.14 total 1324.57 last 63.83) - open/close 47281 (time 1785224061.14 total 1334.58 last 70.04) - open/close 48033 (time 1785224071.15 total 1344.59 last 75.13) - open/close 48714 (time 1785224081.16 total 1354.60 last 68.04) - open/close 49380 (time 1785224091.17 total 1364.61 last 66.55) total: 50000 open/close in 1373.19 seconds: 36.41 ops/second - unlinked 0 (time 1785224103 ; total 0 ; last 0) - unlinked 10000 (time 1785224230 ; total 127 ; last 127) - unlinked 20000 (time 1785224343 ; total 240 ; last 113) - unlinked 30000 (time 1785224441 ; total 338 ; last 98) - unlinked 40000 (time 1785224524 ; total 421 ; last 83) total: 50000 unlinks in 476 seconds: 105.042015 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg121-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg121-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg121-server stop mds service on oleg121-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg121-server unloading modules via unload_modules_local on: 'oleg121-server' oleg121-server: oleg121-server.virtnet: executing unload_modules_local oleg121-server: modules unloaded. pdsh@oleg121-client: oleg121-server: ssh exited with exit code 2 pdsh@oleg121-client: oleg121-client: ssh exited with exit code 2 pdsh@oleg121-client: oleg121-client: ssh exited with exit code 2 pdsh@oleg121-client: oleg121-server: ssh exited with exit code 2