== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 19:53:00 (1789084380) start mds service on oleg253-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg253-server' oleg253-server: oleg253-server.virtnet: executing load_modules_local oleg253-server: Loading modules from /home/green/git/lustre-release/lustre oleg253-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg253-server: detected 4 online CPUs by sysfs oleg253-server: MODOPTS_LIBCFS= oleg253-server: Force libcfs to create 2 CPU partitions oleg253-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg253-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg253-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg253-server: oleg253-server.virtnet: executing set_default_debug -1 all pdsh@oleg253-client: oleg253-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg253-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg253-server: oleg253-server.virtnet: executing set_default_debug -1 all pdsh@oleg253-client: oleg253-server: ssh exited with exit code 1 Started lustre-OST0000 oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg253-client.virtnet: -o user_xattr,flock 192.168.202.153@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 234 (time 1789084464.50 total 10.26 last 22.80) - open/close 458 (time 1789084474.51 total 20.26 last 22.40) - open/close 692 (time 1789084484.51 total 30.27 last 23.38) - open/close 910 (time 1789084494.71 total 40.47 last 21.38) - open/close 1110 (time 1789084505.65 total 51.41 last 18.28) - open/close 1238 (time 1789084515.92 total 61.68 last 12.47) - open/close 1409 (time 1789084525.92 total 71.68 last 17.10) - open/close 1675 (time 1789084535.94 total 81.70 last 26.53) - open/close 1824 (time 1789084545.96 total 91.72 last 14.88) - open/close 2028 (time 1789084555.97 total 101.73 last 20.38) - open/close 2313 (time 1789084566.34 total 112.10 last 27.48) - open/close 2450 (time 1789084576.34 total 122.10 last 13.70) - open/close 2702 (time 1789084586.89 total 132.65 last 23.89) - open/close 2948 (time 1789084597.43 total 143.19 last 23.35) - open/close 3198 (time 1789084608.56 total 154.32 last 22.46) - open/close 3369 (time 1789084619.27 total 165.03 last 15.96) - open/close 3583 (time 1789084629.83 total 175.59 last 20.26) - open/close 3664 (time 1789084641.17 total 186.93 last 7.15) - open/close 3770 (time 1789084651.22 total 196.98 last 10.55) - open/close 4037 (time 1789084661.22 total 206.98 last 26.68) - open/close 4238 (time 1789084671.24 total 217.00 last 20.07) - open/close 4496 (time 1789084681.24 total 227.00 last 25.79) - open/close 4592 (time 1789084691.58 total 237.34 last 9.28) - open/close 4706 (time 1789084701.61 total 247.37 last 11.38) - open/close 4959 (time 1789084711.93 total 257.69 last 24.51) - open/close 5205 (time 1789084722.24 total 268.00 last 23.85) - open/close 5450 (time 1789084732.41 total 278.17 last 24.10) - open/close 5626 (time 1789084742.72 total 288.48 last 17.06) - open/close 5796 (time 1789084753.29 total 299.05 last 16.09) - open/close 6036 (time 1789084763.92 total 309.68 last 22.56) - open/close 6299 (time 1789084774.72 total 320.48 last 24.37) - open/close 6542 (time 1789084785.72 total 331.48 last 22.09) - open/close 6756 (time 1789084798.03 total 343.79 last 17.38) - open/close 6973 (time 1789084808.03 total 353.79 last 21.70) - open/close 7145 (time 1789084818.32 total 364.08 last 16.72) - open/close 7254 (time 1789084829.26 total 375.02 last 9.97) - open/close 7463 (time 1789084839.26 total 385.02 last 20.90) - open/close 7553 (time 1789084849.50 total 395.26 last 8.79) - open/close 7718 (time 1789084859.52 total 405.28 last 16.47) - open/close 7937 (time 1789084869.52 total 415.28 last 21.90) - open/close 8066 (time 1789084879.75 total 425.51 last 12.61) - open/close 8325 (time 1789084890.49 total 436.25 last 24.12) - open/close 8505 (time 1789084900.72 total 446.48 last 17.59) - open/close 8746 (time 1789084910.73 total 456.49 last 24.08) - open/close 8993 (time 1789084921.21 total 466.97 last 23.58) - open/close 9219 (time 1789084932.02 total 477.78 last 20.89) - open/close 9447 (time 1789084942.65 total 488.41 last 21.46) - open/close 9652 (time 1789084952.90 total 498.66 last 20.00) - open/close 9771 (time 1789084963.39 total 509.15 last 11.34) - open/close 9837 (time 1789084973.49 total 519.25 last 6.54) - open/close 10000 (time 1789084979.70 total 525.46 last 26.23) - open/close 10181 (time 1789084990.37 total 536.13 last 16.97) - open/close 10432 (time 1789085000.38 total 546.14 last 25.08) - open/close 10648 (time 1789085010.38 total 556.14 last 21.59) - open/close 10841 (time 1789085020.41 total 566.16 last 19.25) - open/close 11077 (time 1789085030.41 total 576.17 last 23.58) - open/close 11271 (time 1789085042.50 total 588.26 last 16.05) - open/close 11392 (time 1789085053.35 total 599.11 last 11.15) - open/close 11558 (time 1789085066.25 total 612.01 last 12.87) - open/close 11718 (time 1789085076.26 total 622.02 last 15.98) - open/close 11895 (time 1789085087.10 total 632.86 last 16.33) - open/close 11959 (time 1789085097.13 total 642.89 last 6.38) - open/close 12072 (time 1789085107.97 total 653.73 last 10.42) - open/close 12288 (time 1789085119.96 total 665.72 last 18.02) - open/close 12528 (time 1789085129.98 total 675.74 last 23.94) - open/close 12663 (time 1789085140.01 total 685.77 last 13.46) - open/close 12855 (time 1789085150.80 total 696.56 last 17.80) - open/close 13121 (time 1789085160.80 total 706.56 last 26.58) - open/close 13335 (time 1789085170.83 total 716.59 last 21.34) - open/close 13578 (time 1789085180.84 total 726.60 last 24.28) - open/close 13744 (time 1789085190.85 total 736.61 last 16.59) - open/close 13943 (time 1789085201.05 total 746.81 last 19.51) - open/close 14161 (time 1789085211.05 total 756.81 last 21.80) - open/close 14390 (time 1789085221.63 total 767.39 last 21.64) - open/close 14562 (time 1789085231.93 total 777.69 last 16.70) - open/close 14768 (time 1789085242.10 total 787.86 last 20.26) - open/close 15003 (time 1789085252.94 total 798.70 last 21.67) - open/close 15235 (time 1789085263.68 total 809.44 last 21.61) - open/close 15443 (time 1789085274.36 total 820.12 last 19.47) - open/close 15547 (time 1789085284.53 total 830.29 last 10.22) - open/close 15793 (time 1789085294.56 total 840.32 last 24.54) - open/close 15992 (time 1789085306.64 total 852.40 last 16.46) - open/close 16241 (time 1789085317.56 total 863.32 last 22.80) - open/close 16424 (time 1789085327.58 total 873.34 last 18.27) - open/close 16534 (time 1789085338.52 total 884.28 last 10.05) - open/close 16767 (time 1789085349.06 total 894.82 last 22.10) - open/close 16952 (time 1789085360.02 total 905.78 last 16.88) - open/close 17033 (time 1789085370.06 total 915.82 last 8.07) - open/close 17210 (time 1789085380.07 total 925.83 last 17.70) - open/close 17411 (time 1789085390.10 total 935.86 last 20.03) - open/close 17563 (time 1789085400.87 total 946.63 last 14.12) - open/close 17677 (time 1789085411.93 total 957.69 last 10.31) - open/close 17772 (time 1789085422.79 total 968.55 last 8.75) - open/close 17860 (time 1789085433.23 total 978.99 last 8.43) - open/close 18194 (time 1789085443.24 total 989.00 last 33.38) - open/close 19054 (time 1789085453.25 total 999.01 last 85.87) - open/close 19911 (time 1789085463.27 total 1009.02 last 85.58) - open/close 20891 (time 1789085473.28 total 1019.04 last 97.87) - open/close 21809 (time 1789085483.29 total 1029.05 last 91.71) - open/close 22774 (time 1789085493.29 total 1039.05 last 96.50) - open/close 23792 (time 1789085503.29 total 1049.05 last 101.76) - open/close 24799 (time 1789085513.30 total 1059.06 last 100.61) - open/close 25796 (time 1789085523.30 total 1069.06 last 99.69) - open/close 26641 (time 1789085533.31 total 1079.07 last 84.46) - open/close 27600 (time 1789085543.32 total 1089.08 last 95.74) - open/close 28487 (time 1789085553.33 total 1099.09 last 88.65) - open/close 29532 (time 1789085563.33 total 1109.09 last 104.50) - open/close 30000 (time 1789085568.80 total 1114.56 last 85.51) - open/close 30822 (time 1789085578.81 total 1124.57 last 82.13) - open/close 31537 (time 1789085588.82 total 1134.58 last 71.43) - open/close 32371 (time 1789085598.82 total 1144.58 last 83.40) - open/close 33257 (time 1789085608.83 total 1154.59 last 88.55) - open/close 34056 (time 1789085618.85 total 1164.61 last 79.75) - open/close 34924 (time 1789085628.85 total 1174.61 last 86.73) - open/close 35696 (time 1789085638.86 total 1184.62 last 77.18) - open/close 36582 (time 1789085648.86 total 1194.62 last 88.53) - open/close 37540 (time 1789085658.87 total 1204.63 last 95.77) - open/close 38627 (time 1789085668.88 total 1214.63 last 108.62) - open/close 39598 (time 1789085678.88 total 1224.64 last 97.03) - open/close 40000 (time 1789085682.80 total 1228.56 last 102.64) - open/close 40910 (time 1789085692.80 total 1238.56 last 91.00) - open/close 41771 (time 1789085702.81 total 1248.57 last 86.03) - open/close 42714 (time 1789085712.83 total 1258.59 last 94.11) - open/close 43516 (time 1789085722.83 total 1268.59 last 80.19) - open/close 44208 (time 1789085732.89 total 1278.65 last 68.75) - open/close 44959 (time 1789085742.90 total 1288.65 last 75.10) - open/close 45833 (time 1789085752.90 total 1298.66 last 87.34) - open/close 46659 (time 1789085762.91 total 1308.67 last 82.53) - open/close 47504 (time 1789085772.94 total 1318.70 last 84.22) - open/close 48402 (time 1789085782.96 total 1328.72 last 89.70) - open/close 49291 (time 1789085792.96 total 1338.72 last 88.87) total: 50000 open/close in 1346.74 seconds: 37.13 ops/second - unlinked 0 (time 1789085805 ; total 0 ; last 0) - unlinked 10000 (time 1789085931 ; total 126 ; last 126) - unlinked 20000 (time 1789086044 ; total 239 ; last 113) - unlinked 30000 (time 1789086113 ; total 308 ; last 69) - unlinked 40000 (time 1789086190 ; total 385 ; last 77) total: 50000 unlinks in 457 seconds: 109.409187 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg253-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg253-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg253-server stop mds service on oleg253-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg253-server unloading modules via unload_modules_local on: 'oleg253-server' oleg253-server: oleg253-server.virtnet: executing unload_modules_local oleg253-server: modules unloaded. pdsh@oleg253-client: oleg253-server: ssh exited with exit code 2 pdsh@oleg253-client: oleg253-client: ssh exited with exit code 2 pdsh@oleg253-client: oleg253-server: ssh exited with exit code 2 pdsh@oleg253-client: oleg253-client: ssh exited with exit code 2