== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:05:18 (1788156318) start mds service on oleg433-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg433-server' oleg433-server: oleg433-server.virtnet: executing load_modules_local oleg433-server: Loading modules from /home/green/git/lustre-release/lustre oleg433-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg433-server: detected 4 online CPUs by sysfs oleg433-server: MODOPTS_LIBCFS= oleg433-server: Force libcfs to create 2 CPU partitions oleg433-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg433-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg433-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg433-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Started lustre-OST0000 oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg433-client.virtnet: -o user_xattr,flock 192.168.204.133@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 235 (time 1788156406.92 total 12.93 last 18.17) - open/close 502 (time 1788156416.93 total 22.94 last 26.67) - open/close 736 (time 1788156427.14 total 33.15 last 22.92) - open/close 977 (time 1788156437.14 total 43.15 last 24.09) - open/close 1159 (time 1788156447.18 total 53.20 last 18.12) - open/close 1401 (time 1788156457.18 total 63.20 last 24.20) - open/close 1561 (time 1788156467.19 total 73.20 last 15.99) - open/close 1811 (time 1788156477.49 total 83.51 last 24.26) - open/close 2031 (time 1788156487.52 total 93.53 last 21.95) - open/close 2197 (time 1788156498.06 total 104.07 last 15.75) - open/close 2446 (time 1788156508.07 total 114.09 last 24.85) - open/close 2504 (time 1788156518.34 total 124.36 last 5.65) - open/close 2713 (time 1788156528.35 total 134.37 last 20.88) - open/close 2824 (time 1788156538.68 total 144.70 last 10.75) - open/close 3041 (time 1788156549.06 total 155.07 last 20.92) - open/close 3199 (time 1788156559.95 total 165.97 last 14.50) - open/close 3389 (time 1788156569.96 total 175.98 last 18.98) - open/close 3631 (time 1788156579.96 total 185.98 last 24.20) - open/close 3841 (time 1788156592.51 total 198.52 last 16.74) - open/close 4091 (time 1788156602.53 total 208.54 last 24.96) - open/close 4280 (time 1788156613.05 total 219.06 last 17.96) - open/close 4512 (time 1788156623.29 total 229.31 last 22.65) - open/close 4744 (time 1788156633.32 total 239.34 last 23.13) - open/close 4906 (time 1788156643.71 total 249.72 last 15.60) - open/close 5123 (time 1788156653.87 total 259.88 last 21.35) - open/close 5349 (time 1788156664.45 total 270.46 last 21.37) - open/close 5563 (time 1788156674.95 total 280.96 last 20.38) - open/close 5829 (time 1788156685.19 total 291.20 last 25.97) - open/close 6036 (time 1788156696.00 total 302.01 last 19.15) - open/close 6242 (time 1788156706.87 total 312.88 last 18.95) - open/close 6412 (time 1788156719.80 total 325.82 last 13.14) - open/close 6577 (time 1788156729.83 total 335.84 last 16.46) - open/close 6797 (time 1788156740.68 total 346.69 last 20.27) - open/close 6937 (time 1788156751.58 total 357.60 last 12.84) - open/close 7095 (time 1788156761.59 total 367.60 last 15.79) - open/close 7292 (time 1788156771.59 total 377.61 last 19.69) - open/close 7504 (time 1788156781.59 total 387.61 last 21.20) - open/close 7680 (time 1788156792.27 total 398.28 last 16.49) - open/close 7872 (time 1788156802.50 total 408.51 last 18.76) - open/close 8037 (time 1788156812.50 total 418.52 last 16.50) - open/close 8215 (time 1788156822.55 total 428.56 last 17.71) - open/close 8382 (time 1788156832.58 total 438.60 last 16.65) - open/close 8550 (time 1788156843.23 total 449.24 last 15.78) - open/close 8717 (time 1788156853.23 total 459.25 last 16.69) - open/close 8853 (time 1788156863.26 total 469.27 last 13.56) - open/close 9010 (time 1788156873.59 total 479.60 last 15.20) - open/close 9227 (time 1788156883.77 total 489.78 last 21.32) - open/close 9484 (time 1788156893.77 total 499.78 last 25.70) - open/close 9579 (time 1788156905.20 total 511.22 last 8.31) - open/close 9740 (time 1788156915.22 total 521.24 last 16.06) - open/close 9852 (time 1788156925.95 total 531.96 last 10.45) - open/close 9951 (time 1788156936.38 total 542.39 last 9.49) - open/close 10102 (time 1788156946.96 total 552.97 last 14.27) - open/close 10267 (time 1788156957.57 total 563.59 last 15.55) - open/close 10390 (time 1788156967.75 total 573.76 last 12.09) - open/close 10505 (time 1788156977.97 total 583.98 last 11.25) - open/close 10658 (time 1788156987.97 total 593.98 last 15.30) - open/close 10751 (time 1788156998.98 total 605.00 last 8.45) - open/close 10880 (time 1788157009.51 total 615.53 last 12.25) - open/close 11002 (time 1788157020.61 total 626.62 last 11.00) - open/close 11149 (time 1788157033.15 total 639.17 last 11.72) - open/close 11271 (time 1788157043.16 total 649.17 last 12.19) - open/close 11465 (time 1788157053.17 total 659.18 last 19.38) - open/close 11619 (time 1788157063.79 total 669.80 last 14.50) - open/close 11767 (time 1788157074.26 total 680.28 last 14.13) - open/close 11868 (time 1788157084.86 total 690.87 last 9.53) - open/close 12013 (time 1788157095.94 total 701.96 last 13.08) - open/close 12121 (time 1788157108.22 total 714.24 last 8.80) - open/close 12307 (time 1788157118.23 total 724.25 last 18.58) - open/close 12481 (time 1788157128.97 total 734.98 last 16.21) - open/close 12597 (time 1788157138.98 total 745.00 last 11.58) - open/close 12748 (time 1788157149.90 total 755.92 last 13.83) - open/close 12975 (time 1788157159.94 total 765.95 last 22.62) - open/close 13171 (time 1788157169.96 total 775.97 last 19.56) - open/close 13354 (time 1788157180.55 total 786.56 last 17.27) - open/close 13420 (time 1788157190.71 total 796.73 last 6.49) - open/close 13614 (time 1788157201.40 total 807.42 last 18.15) - open/close 13733 (time 1788157211.78 total 817.79 last 11.47) - open/close 13831 (time 1788157221.78 total 827.79 last 9.80) - open/close 13861 (time 1788157231.96 total 837.98 last 2.95) - open/close 13975 (time 1788157241.99 total 848.00 last 11.38) - open/close 14111 (time 1788157252.57 total 858.58 last 12.85) - open/close 14197 (time 1788157263.25 total 869.27 last 8.05) - open/close 14301 (time 1788157273.73 total 879.75 last 9.92) - open/close 14416 (time 1788157284.30 total 890.32 last 10.88) - open/close 14581 (time 1788157294.80 total 900.81 last 15.73) - open/close 14690 (time 1788157305.63 total 911.64 last 10.06) - open/close 14809 (time 1788157315.66 total 921.68 last 11.85) - open/close 14847 (time 1788157326.08 total 932.09 last 3.65) - open/close 15043 (time 1788157336.71 total 942.72 last 18.44) - open/close 15218 (time 1788157346.71 total 952.73 last 17.50) - open/close 15391 (time 1788157356.87 total 962.88 last 17.03) - open/close 15568 (time 1788157367.10 total 973.12 last 17.29) - open/close 15713 (time 1788157377.78 total 983.80 last 13.58) - open/close 15839 (time 1788157388.53 total 994.55 last 11.72) - open/close 16084 (time 1788157398.54 total 1004.55 last 24.49) - open/close 16305 (time 1788157408.65 total 1014.66 last 21.86) - open/close 16429 (time 1788157418.67 total 1024.68 last 12.38) - open/close 16924 (time 1788157428.67 total 1034.69 last 49.47) - open/close 17517 (time 1788157438.68 total 1044.69 last 59.26) - open/close 18114 (time 1788157448.68 total 1054.70 last 59.69) - open/close 18689 (time 1788157458.69 total 1064.70 last 57.46) - open/close 19281 (time 1788157468.70 total 1074.71 last 59.14) - open/close 19833 (time 1788157478.72 total 1084.73 last 55.09) - open/close 20000 (time 1788157481.73 total 1087.75 last 55.40) - open/close 20534 (time 1788157491.74 total 1097.75 last 53.36) - open/close 21143 (time 1788157501.75 total 1107.76 last 60.87) - open/close 21690 (time 1788157511.75 total 1117.76 last 54.67) - open/close 22288 (time 1788157521.78 total 1127.79 last 59.65) - open/close 22856 (time 1788157531.81 total 1137.82 last 56.63) - open/close 23420 (time 1788157541.81 total 1147.83 last 56.36) - open/close 23975 (time 1788157551.82 total 1157.83 last 55.47) - open/close 24587 (time 1788157561.82 total 1167.84 last 61.17) - open/close 25156 (time 1788157571.84 total 1177.85 last 56.82) - open/close 25702 (time 1788157581.84 total 1187.85 last 54.58) - open/close 26298 (time 1788157591.85 total 1197.86 last 59.54) - open/close 26868 (time 1788157601.86 total 1207.87 last 56.95) - open/close 27462 (time 1788157611.87 total 1217.88 last 59.35) - open/close 28063 (time 1788157621.89 total 1227.91 last 59.93) - open/close 28647 (time 1788157631.90 total 1237.91 last 58.39) - open/close 29252 (time 1788157641.90 total 1247.92 last 60.46) - open/close 29881 (time 1788157651.91 total 1257.92 last 62.87) - open/close 30000 (time 1788157653.96 total 1259.97 last 58.12) - open/close 30571 (time 1788157663.96 total 1269.98 last 57.05) - open/close 31184 (time 1788157673.97 total 1279.99 last 61.24) - open/close 31794 (time 1788157683.98 total 1289.99 last 60.98) - open/close 32420 (time 1788157693.98 total 1300.00 last 62.57) - open/close 33039 (time 1788157703.99 total 1310.01 last 61.85) - open/close 33609 (time 1788157713.99 total 1320.01 last 56.98) - open/close 34246 (time 1788157724.01 total 1330.02 last 63.62) - open/close 34892 (time 1788157734.01 total 1340.03 last 64.57) - open/close 35474 (time 1788157744.01 total 1350.03 last 58.19) - open/close 36050 (time 1788157754.01 total 1360.03 last 57.59) - open/close 36650 (time 1788157764.02 total 1370.03 last 59.98) - open/close 37211 (time 1788157774.03 total 1380.05 last 56.03) - open/close 37783 (time 1788157784.04 total 1390.06 last 57.14) - open/close 38319 (time 1788157794.07 total 1400.08 last 53.47) - open/close 38892 (time 1788157804.10 total 1410.11 last 57.13) - open/close 39383 (time 1788157814.12 total 1420.13 last 48.99) - open/close 39971 (time 1788157824.14 total 1430.15 last 58.70) - open/close 40491 (time 1788157834.15 total 1440.17 last 51.92) - open/close 41048 (time 1788157844.17 total 1450.19 last 55.58) - open/close 41643 (time 1788157854.18 total 1460.19 last 59.47) - open/close 42265 (time 1788157864.18 total 1470.20 last 62.17) - open/close 42884 (time 1788157874.19 total 1480.21 last 61.83) - open/close 43482 (time 1788157884.21 total 1490.22 last 59.73) - open/close 44046 (time 1788157894.21 total 1500.22 last 56.39) - open/close 44666 (time 1788157904.22 total 1510.23 last 61.94) - open/close 45317 (time 1788157914.22 total 1520.23 last 65.10) - open/close 45944 (time 1788157924.22 total 1530.24 last 62.67) - open/close 46517 (time 1788157934.23 total 1540.24 last 57.28) - open/close 47156 (time 1788157944.24 total 1550.25 last 63.82) - open/close 47809 (time 1788157954.25 total 1560.26 last 65.26) - open/close 48483 (time 1788157964.28 total 1570.29 last 67.20) - open/close 49133 (time 1788157974.28 total 1580.30 last 64.96) - open/close 49746 (time 1788157984.29 total 1590.30 last 61.27) total: 50000 open/close in 1594.52 seconds: 31.36 ops/second - unlinked 0 (time 1788157993 ; total 0 ; last 0) - unlinked 10000 (time 1788158172 ; total 179 ; last 179) - unlinked 20000 (time 1788158351 ; total 358 ; last 179) - unlinked 30000 (time 1788158524 ; total 531 ; last 173) - unlinked 40000 (time 1788158699 ; total 706 ; last 175) total: 50000 unlinks in 881 seconds: 56.753689 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg433-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg433-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg433-server stop mds service on oleg433-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg433-server unloading modules via unload_modules_local on: 'oleg433-server' oleg433-server: oleg433-server.virtnet: executing unload_modules_local oleg433-server: modules unloaded. pdsh@oleg433-client: oleg433-client: ssh exited with exit code 2 pdsh@oleg433-client: oleg433-server: ssh exited with exit code 2 pdsh@oleg433-client: oleg433-server: ssh exited with exit code 2 pdsh@oleg433-client: oleg433-client: ssh exited with exit code 2