== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 15:41:50 (1777578110) start mds service on oleg438-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing load_modules_local oleg438-server: Loading modules from /home/green/git/lustre-release/lustre oleg438-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg438-server: detected 4 online CPUs by sysfs oleg438-server: MODOPTS_LIBCFS= oleg438-server: Force libcfs to create 2 CPU partitions oleg438-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg438-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg438-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg438-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg438-client.virtnet: -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 219 (time 1777578193.73 total 10.20 last 21.47) - open/close 461 (time 1777578204.80 total 21.27 last 21.86) - open/close 690 (time 1777578214.81 total 31.28 last 22.88) - open/close 893 (time 1777578225.14 total 41.61 last 19.66) - open/close 1042 (time 1777578235.38 total 51.85 last 14.55) - open/close 1184 (time 1777578247.83 total 64.30 last 11.41) - open/close 1318 (time 1777578257.85 total 74.32 last 13.37) - open/close 1444 (time 1777578269.31 total 85.78 last 11.00) - open/close 1488 (time 1777578280.02 total 96.50 last 4.11) - open/close 1623 (time 1777578290.03 total 106.50 last 13.50) - open/close 1674 (time 1777578301.71 total 118.18 last 4.36) - open/close 1802 (time 1777578312.13 total 128.60 last 12.28) - open/close 1996 (time 1777578322.17 total 138.64 last 19.33) - open/close 2148 (time 1777578334.97 total 151.45 last 11.87) - open/close 2235 (time 1777578345.67 total 162.14 last 8.14) - open/close 2374 (time 1777578356.36 total 172.83 last 13.00) - open/close 2436 (time 1777578366.36 total 182.84 last 6.20) - open/close 2508 (time 1777578376.71 total 193.18 last 6.96) - open/close 2626 (time 1777578386.75 total 203.22 last 11.75) - open/close 2746 (time 1777578396.75 total 213.23 last 11.99) - open/close 2951 (time 1777578407.24 total 223.72 last 19.54) - open/close 3163 (time 1777578421.83 total 238.31 last 14.53) - open/close 3221 (time 1777578433.75 total 250.22 last 4.87) - open/close 3299 (time 1777578444.28 total 260.75 last 7.41) - open/close 3394 (time 1777578455.16 total 271.63 last 8.73) - open/close 3597 (time 1777578465.92 total 282.39 last 18.87) - open/close 3802 (time 1777578475.94 total 292.41 last 20.46) - open/close 3990 (time 1777578487.29 total 303.77 last 16.55) - open/close 4243 (time 1777578497.31 total 313.78 last 25.27) - open/close 4326 (time 1777578507.38 total 323.86 last 8.24) - open/close 4535 (time 1777578519.31 total 335.78 last 17.53) - open/close 4718 (time 1777578529.33 total 345.80 last 18.26) - open/close 4821 (time 1777578539.82 total 356.29 last 9.82) - open/close 4958 (time 1777578549.84 total 366.32 last 13.66) - open/close 5070 (time 1777578559.90 total 376.37 last 11.14) - open/close 5216 (time 1777578569.93 total 386.40 last 14.55) - open/close 5356 (time 1777578580.74 total 397.21 last 12.96) - open/close 5626 (time 1777578593.59 total 410.07 last 21.00) - open/close 5833 (time 1777578604.10 total 420.57 last 19.71) - open/close 6048 (time 1777578614.11 total 430.58 last 21.47) - open/close 6111 (time 1777578624.12 total 440.59 last 6.29) - open/close 6197 (time 1777578634.77 total 451.25 last 8.07) - open/close 6282 (time 1777578645.28 total 461.75 last 8.09) - open/close 6358 (time 1777578657.95 total 474.42 last 6.00) - open/close 6472 (time 1777578668.51 total 484.99 last 10.79) - open/close 6590 (time 1777578678.67 total 495.14 last 11.62) - open/close 6729 (time 1777578689.65 total 506.12 last 12.66) - open/close 6887 (time 1777578699.97 total 516.44 last 15.31) - open/close 7167 (time 1777578713.02 total 529.50 last 21.45) - open/close 7324 (time 1777578723.39 total 539.86 last 15.15) - open/close 7434 (time 1777578736.07 total 552.54 last 8.67) - open/close 7525 (time 1777578746.09 total 562.56 last 9.08) - open/close 7662 (time 1777578757.20 total 573.67 last 12.33) - open/close 7727 (time 1777578767.20 total 583.67 last 6.50) - open/close 7884 (time 1777578777.22 total 593.69 last 15.67) - open/close 7957 (time 1777578787.29 total 603.77 last 7.25) - open/close 8028 (time 1777578797.76 total 614.23 last 6.78) - open/close 8154 (time 1777578807.79 total 624.26 last 12.57) - open/close 8344 (time 1777578817.92 total 634.39 last 18.74) - open/close 8392 (time 1777578828.29 total 644.76 last 4.63) - open/close 8537 (time 1777578840.89 total 657.36 last 11.51) - open/close 8666 (time 1777578851.31 total 667.78 last 12.38) - open/close 8767 (time 1777578862.34 total 678.81 last 9.16) - open/close 8899 (time 1777578873.30 total 689.77 last 12.04) - open/close 8957 (time 1777578884.22 total 700.70 last 5.31) - open/close 9070 (time 1777578894.23 total 710.70 last 11.29) - open/close 9241 (time 1777578904.24 total 720.71 last 17.08) - open/close 9396 (time 1777578916.13 total 732.60 last 13.04) - open/close 9547 (time 1777578926.42 total 742.89 last 14.67) - open/close 9698 (time 1777578937.46 total 753.93 last 13.67) - open/close 9854 (time 1777578948.07 total 764.55 last 14.70) - open/close 10000 (time 1777578956.77 total 773.25 last 16.78) - open/close 10101 (time 1777578968.51 total 784.99 last 8.60) - open/close 10278 (time 1777578978.52 total 794.99 last 17.70) - open/close 10466 (time 1777578988.55 total 805.02 last 18.74) - open/close 10596 (time 1777578998.90 total 815.37 last 12.55) - open/close 10647 (time 1777579010.71 total 827.18 last 4.32) - open/close 10753 (time 1777579022.80 total 839.27 last 8.77) - open/close 10859 (time 1777579033.19 total 849.66 last 10.20) - open/close 10989 (time 1777579043.22 total 859.69 last 12.96) - open/close 11113 (time 1777579053.57 total 870.04 last 11.98) - open/close 11253 (time 1777579064.29 total 880.76 last 13.06) - open/close 11440 (time 1777579075.15 total 891.62 last 17.21) - open/close 11544 (time 1777579085.17 total 901.64 last 10.38) - open/close 11669 (time 1777579095.17 total 911.65 last 12.49) - open/close 11848 (time 1777579105.19 total 921.66 last 17.87) - open/close 11897 (time 1777579115.83 total 932.31 last 4.60) - open/close 12057 (time 1777579125.94 total 942.41 last 15.84) - open/close 12312 (time 1777579136.46 total 952.93 last 24.25) - open/close 12437 (time 1777579146.47 total 962.94 last 12.49) - open/close 12670 (time 1777579156.49 total 972.96 last 23.25) - open/close 12855 (time 1777579166.56 total 983.03 last 18.37) - open/close 12910 (time 1777579177.63 total 994.10 last 4.97) - open/close 13047 (time 1777579187.63 total 1004.10 last 13.70) - open/close 13100 (time 1777579198.15 total 1014.62 last 5.04) - open/close 13247 (time 1777579208.18 total 1024.65 last 14.66) - open/close 13342 (time 1777579218.94 total 1035.41 last 8.83) - open/close 13413 (time 1777579229.12 total 1045.60 last 6.97) - open/close 13440 (time 1777579239.16 total 1055.64 last 2.69) - open/close 13604 (time 1777579249.40 total 1065.87 last 16.02) - open/close 13847 (time 1777579259.49 total 1075.96 last 24.08) - open/close 13996 (time 1777579270.93 total 1087.40 last 13.02) - open/close 14096 (time 1777579280.96 total 1097.43 last 9.97) - open/close 14538 (time 1777579290.97 total 1107.44 last 44.17) - open/close 15074 (time 1777579300.99 total 1117.46 last 53.48) - open/close 15699 (time 1777579310.99 total 1127.46 last 62.49) - open/close 16267 (time 1777579321.00 total 1137.47 last 56.76) - open/close 16821 (time 1777579331.01 total 1147.49 last 55.31) - open/close 17334 (time 1777579341.03 total 1157.50 last 51.23) - open/close 17740 (time 1777579351.06 total 1167.53 last 40.48) - open/close 18241 (time 1777579361.06 total 1177.53 last 50.08) - open/close 18822 (time 1777579371.07 total 1187.54 last 58.04) - open/close 19401 (time 1777579381.09 total 1197.56 last 57.80) - open/close 19963 (time 1777579391.11 total 1207.58 last 56.08) - open/close 20541 (time 1777579401.11 total 1217.59 last 57.77) - open/close 21151 (time 1777579411.13 total 1227.60 last 60.91) - open/close 21715 (time 1777579421.14 total 1237.61 last 56.32) - open/close 22306 (time 1777579431.17 total 1247.65 last 58.92) - open/close 22843 (time 1777579441.18 total 1257.65 last 53.67) - open/close 23401 (time 1777579451.19 total 1267.66 last 55.75) - open/close 23983 (time 1777579461.19 total 1277.67 last 58.16) - open/close 24542 (time 1777579471.20 total 1287.67 last 55.89) - open/close 25093 (time 1777579481.22 total 1297.69 last 54.96) - open/close 25656 (time 1777579491.23 total 1307.70 last 56.28) - open/close 26281 (time 1777579501.23 total 1317.70 last 62.45) - open/close 26887 (time 1777579511.24 total 1327.71 last 60.56) - open/close 27500 (time 1777579521.25 total 1337.72 last 61.24) - open/close 28088 (time 1777579531.26 total 1347.73 last 58.77) - open/close 28669 (time 1777579541.26 total 1357.73 last 58.09) - open/close 29252 (time 1777579551.26 total 1367.74 last 58.26) - open/close 29845 (time 1777579561.27 total 1377.74 last 59.25) - open/close 30000 (time 1777579563.81 total 1380.28 last 61.02) - open/close 30503 (time 1777579573.82 total 1390.29 last 50.28) - open/close 31093 (time 1777579583.84 total 1400.31 last 58.88) - open/close 31645 (time 1777579593.84 total 1410.31 last 55.20) - open/close 32161 (time 1777579603.84 total 1420.31 last 51.59) - open/close 32685 (time 1777579613.85 total 1430.32 last 52.35) - open/close 33191 (time 1777579623.86 total 1440.33 last 50.54) - open/close 33692 (time 1777579633.87 total 1450.34 last 50.05) - open/close 34293 (time 1777579643.88 total 1460.35 last 60.06) - open/close 34822 (time 1777579653.88 total 1470.36 last 52.87) - open/close 35400 (time 1777579663.89 total 1480.36 last 57.78) - open/close 35873 (time 1777579673.89 total 1490.36 last 47.30) - open/close 36423 (time 1777579683.93 total 1500.40 last 54.80) - open/close 36850 (time 1777579693.94 total 1510.41 last 42.63) - open/close 37438 (time 1777579703.95 total 1520.42 last 58.77) - open/close 38090 (time 1777579713.95 total 1530.42 last 65.17) - open/close 38625 (time 1777579723.96 total 1540.44 last 53.44) - open/close 39115 (time 1777579733.97 total 1550.44 last 48.97) - open/close 39588 (time 1777579743.99 total 1560.46 last 47.22) - open/close 40000 (time 1777579753.82 total 1570.30 last 41.88) - open/close 40433 (time 1777579763.83 total 1580.30 last 43.29) - open/close 40865 (time 1777579773.85 total 1590.33 last 43.08) - open/close 41391 (time 1777579783.89 total 1600.36 last 52.43) - open/close 41912 (time 1777579793.90 total 1610.37 last 52.04) - open/close 42463 (time 1777579803.91 total 1620.38 last 55.02) - open/close 43011 (time 1777579813.94 total 1630.41 last 54.67) - open/close 43546 (time 1777579823.94 total 1640.41 last 53.47) - open/close 44090 (time 1777579833.95 total 1650.42 last 54.33) - open/close 44615 (time 1777579843.97 total 1660.44 last 52.42) - open/close 45158 (time 1777579853.98 total 1670.45 last 54.23) - open/close 45651 (time 1777579863.99 total 1680.46 last 49.24) - open/close 46167 (time 1777579874.01 total 1690.48 last 51.53) - open/close 46665 (time 1777579884.01 total 1700.48 last 49.78) - open/close 47164 (time 1777579894.01 total 1710.48 last 49.90) - open/close 47696 (time 1777579904.03 total 1720.50 last 53.12) - open/close 48255 (time 1777579914.03 total 1730.50 last 55.90) - open/close 48816 (time 1777579924.03 total 1740.51 last 56.06) - open/close 49349 (time 1777579934.05 total 1750.52 last 53.24) - open/close 49881 (time 1777579944.05 total 1760.53 last 53.15) total: 50000 open/close in 1762.63 seconds: 28.37 ops/second - unlinked 0 (time 1777579950 ; total 0 ; last 0) - unlinked 10000 (time 1777580146 ; total 196 ; last 196) - unlinked 20000 (time 1777580341 ; total 391 ; last 195) - unlinked 30000 (time 1777580542 ; total 592 ; last 201) - unlinked 40000 (time 1777580729 ; total 779 ; last 187) total: 50000 unlinks in 978 seconds: 51.124744 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg438-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg438-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg438-server stop mds service on oleg438-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg438-server unloading modules via unload_modules_local on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing unload_modules_local oleg438-server: modules unloaded. pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-client: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-client: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2