== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 03:45:35 (1784187935) start mds service on oleg448-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg448-server' oleg448-server: oleg448-server.virtnet: executing load_modules_local oleg448-server: Loading modules from /home/green/git/lustre-release/lustre oleg448-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg448-server: detected 4 online CPUs by sysfs oleg448-server: MODOPTS_LIBCFS= oleg448-server: Force libcfs to create 2 CPU partitions oleg448-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg448-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg448-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg448-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-OST0000 oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg448-client.virtnet: -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 130 (time 1784188021.25 total 10.27 last 12.66) - open/close 268 (time 1784188031.88 total 20.90 last 12.98) - open/close 462 (time 1784188041.89 total 30.91 last 19.38) - open/close 696 (time 1784188052.80 total 41.81 last 21.45) - open/close 970 (time 1784188063.62 total 52.64 last 25.32) - open/close 1112 (time 1784188073.93 total 62.94 last 13.78) - open/close 1343 (time 1784188083.94 total 72.96 last 23.08) - open/close 1532 (time 1784188094.25 total 83.27 last 18.32) - open/close 1723 (time 1784188104.99 total 94.01 last 17.78) - open/close 1885 (time 1784188115.01 total 104.03 last 16.17) - open/close 2120 (time 1784188125.68 total 114.70 last 22.03) - open/close 2408 (time 1784188135.69 total 124.71 last 28.78) - open/close 2559 (time 1784188146.28 total 135.30 last 14.25) - open/close 2686 (time 1784188156.58 total 145.60 last 12.33) - open/close 2787 (time 1784188166.72 total 155.74 last 9.97) - open/close 2869 (time 1784188177.33 total 166.35 last 7.73) - open/close 2931 (time 1784188187.89 total 176.91 last 5.87) - open/close 3062 (time 1784188197.92 total 186.94 last 13.07) - open/close 3277 (time 1784188210.54 total 199.56 last 17.03) - open/close 3509 (time 1784188221.43 total 210.45 last 21.31) - open/close 3633 (time 1784188232.28 total 221.29 last 11.43) - open/close 3814 (time 1784188243.33 total 232.35 last 16.37) - open/close 4033 (time 1784188254.26 total 243.27 last 20.04) - open/close 4227 (time 1784188266.23 total 255.25 last 16.20) - open/close 4365 (time 1784188276.35 total 265.37 last 13.63) - open/close 4552 (time 1784188288.23 total 277.25 last 15.75) - open/close 4730 (time 1784188298.25 total 287.26 last 17.77) - open/close 4870 (time 1784188308.46 total 297.48 last 13.70) - open/close 4985 (time 1784188318.51 total 307.53 last 11.44) - open/close 5123 (time 1784188328.88 total 317.90 last 13.31) - open/close 5262 (time 1784188339.51 total 328.53 last 13.08) - open/close 5479 (time 1784188349.52 total 338.54 last 21.68) - open/close 5646 (time 1784188359.54 total 348.56 last 16.66) - open/close 5865 (time 1784188369.57 total 358.59 last 21.83) - open/close 6070 (time 1784188381.63 total 370.65 last 17.00) - open/close 6281 (time 1784188391.98 total 381.00 last 20.39) - open/close 6340 (time 1784188402.68 total 391.70 last 5.51) - open/close 6443 (time 1784188413.59 total 402.61 last 9.44) - open/close 6553 (time 1784188423.69 total 412.71 last 10.90) - open/close 6658 (time 1784188433.70 total 422.72 last 10.49) - open/close 6838 (time 1784188447.16 total 436.18 last 13.37) - open/close 6982 (time 1784188457.86 total 446.88 last 13.46) - open/close 7090 (time 1784188470.06 total 459.08 last 8.85) - open/close 7244 (time 1784188482.83 total 471.85 last 12.06) - open/close 7370 (time 1784188493.86 total 482.88 last 11.42) - open/close 7589 (time 1784188503.86 total 492.88 last 21.90) - open/close 7787 (time 1784188514.36 total 503.37 last 18.87) - open/close 7965 (time 1784188524.55 total 513.57 last 17.45) - open/close 8149 (time 1784188534.57 total 523.59 last 18.37) - open/close 8303 (time 1784188544.58 total 533.60 last 15.38) - open/close 8463 (time 1784188556.58 total 545.60 last 13.33) - open/close 8650 (time 1784188566.59 total 555.61 last 18.69) - open/close 8774 (time 1784188576.60 total 565.61 last 12.39) - open/close 8851 (time 1784188586.60 total 575.62 last 7.69) - open/close 9049 (time 1784188597.55 total 586.57 last 18.08) - open/close 9219 (time 1784188607.56 total 596.58 last 16.99) - open/close 9374 (time 1784188617.86 total 606.88 last 15.05) - open/close 9447 (time 1784188628.65 total 617.66 last 6.77) - open/close 9547 (time 1784188639.23 total 628.25 last 9.44) - open/close 9813 (time 1784188649.77 total 638.79 last 25.24) - open/close 9879 (time 1784188660.04 total 649.06 last 6.43) - open/close 10000 (time 1784188666.57 total 655.59 last 18.53) - open/close 10060 (time 1784188677.64 total 666.66 last 5.42) - open/close 10119 (time 1784188687.67 total 676.69 last 5.88) - open/close 10293 (time 1784188698.30 total 687.32 last 16.37) - open/close 10455 (time 1784188708.32 total 697.34 last 16.17) - open/close 10609 (time 1784188723.48 total 712.49 last 10.16) - open/close 10794 (time 1784188733.48 total 722.50 last 18.50) - open/close 10942 (time 1784188743.99 total 733.01 last 14.08) - open/close 11038 (time 1784188755.77 total 744.79 last 8.15) - open/close 11254 (time 1784188767.84 total 756.86 last 17.89) - open/close 11389 (time 1784188779.13 total 768.15 last 11.96) - open/close 11546 (time 1784188789.82 total 778.84 last 14.69) - open/close 11692 (time 1784188802.56 total 791.58 last 11.46) - open/close 11931 (time 1784188812.57 total 801.58 last 23.89) - open/close 12048 (time 1784188822.58 total 811.59 last 11.69) - open/close 12229 (time 1784188833.13 total 822.15 last 17.14) - open/close 12327 (time 1784188844.11 total 833.13 last 8.92) - open/close 12450 (time 1784188856.65 total 845.67 last 9.81) - open/close 12527 (time 1784188866.77 total 855.78 last 7.61) - open/close 12583 (time 1784188877.70 total 866.72 last 5.12) - open/close 12724 (time 1784188887.70 total 876.72 last 14.09) - open/close 12885 (time 1784188897.71 total 886.73 last 16.08) - open/close 13071 (time 1784188907.90 total 896.91 last 18.27) - open/close 13278 (time 1784188919.92 total 908.94 last 17.21) - open/close 13490 (time 1784188929.93 total 918.95 last 21.19) - open/close 13587 (time 1784188940.02 total 929.04 last 9.61) - open/close 13810 (time 1784188950.99 total 940.01 last 20.32) - open/close 14019 (time 1784188961.84 total 950.86 last 19.27) - open/close 14196 (time 1784188972.79 total 961.81 last 16.16) - open/close 14263 (time 1784188983.81 total 972.83 last 6.08) - open/close 14492 (time 1784188993.83 total 982.85 last 22.87) - open/close 14751 (time 1784189004.53 total 993.55 last 24.20) - open/close 14941 (time 1784189014.53 total 1003.55 last 19.00) - open/close 15167 (time 1784189025.59 total 1014.61 last 20.44) - open/close 15395 (time 1784189035.70 total 1024.72 last 22.56) - open/close 15552 (time 1784189045.71 total 1034.72 last 15.69) - open/close 15865 (time 1784189055.73 total 1044.74 last 31.24) - open/close 16484 (time 1784189065.74 total 1054.76 last 61.80) - open/close 17049 (time 1784189075.75 total 1064.77 last 56.45) - open/close 17610 (time 1784189085.76 total 1074.77 last 56.07) - open/close 18146 (time 1784189095.77 total 1084.79 last 53.52) - open/close 18714 (time 1784189105.78 total 1094.80 last 56.77) - open/close 19315 (time 1784189115.78 total 1104.80 last 60.06) - open/close 19900 (time 1784189125.79 total 1114.81 last 58.46) - open/close 20428 (time 1784189135.81 total 1124.83 last 52.71) - open/close 20986 (time 1784189145.82 total 1134.84 last 55.71) - open/close 21535 (time 1784189155.84 total 1144.86 last 54.81) - open/close 22110 (time 1784189165.84 total 1154.86 last 57.47) - open/close 22682 (time 1784189175.86 total 1164.88 last 57.12) - open/close 23197 (time 1784189185.88 total 1174.90 last 51.40) - open/close 23696 (time 1784189195.88 total 1184.90 last 49.89) - open/close 24233 (time 1784189205.89 total 1194.91 last 53.64) - open/close 24847 (time 1784189215.90 total 1204.92 last 61.36) - open/close 25454 (time 1784189225.91 total 1214.93 last 60.66) - open/close 26080 (time 1784189235.91 total 1224.93 last 62.58) - open/close 26664 (time 1784189245.91 total 1234.93 last 58.38) - open/close 27061 (time 1784189255.91 total 1244.93 last 39.69) - open/close 27583 (time 1784189265.92 total 1254.94 last 52.18) - open/close 28242 (time 1784189275.92 total 1264.94 last 65.88) - open/close 28826 (time 1784189285.93 total 1274.94 last 58.38) - open/close 29422 (time 1784189295.93 total 1284.95 last 59.54) - open/close 29975 (time 1784189305.94 total 1294.96 last 55.27) - open/close 30557 (time 1784189315.96 total 1304.98 last 58.09) - open/close 31122 (time 1784189325.96 total 1314.98 last 56.50) - open/close 31694 (time 1784189335.97 total 1324.99 last 57.13) - open/close 32217 (time 1784189345.98 total 1335.00 last 52.26) - open/close 32803 (time 1784189355.99 total 1345.01 last 58.56) - open/close 33386 (time 1784189365.99 total 1355.01 last 58.26) - open/close 33976 (time 1784189375.99 total 1365.01 last 59.00) - open/close 34544 (time 1784189386.02 total 1375.04 last 56.65) - open/close 35178 (time 1784189396.03 total 1385.05 last 63.37) - open/close 35668 (time 1784189406.04 total 1395.06 last 48.94) - open/close 36233 (time 1784189416.05 total 1405.07 last 56.42) - open/close 36815 (time 1784189426.08 total 1415.09 last 58.08) - open/close 37399 (time 1784189436.09 total 1425.11 last 58.32) - open/close 37948 (time 1784189446.10 total 1435.12 last 54.86) - open/close 38528 (time 1784189456.12 total 1445.14 last 57.85) - open/close 39099 (time 1784189466.14 total 1455.16 last 57.02) - open/close 39649 (time 1784189476.15 total 1465.17 last 54.95) - open/close 40000 (time 1784189482.43 total 1471.45 last 55.83) - open/close 40471 (time 1784189492.44 total 1481.46 last 47.05) - open/close 40940 (time 1784189502.46 total 1491.48 last 46.84) - open/close 41430 (time 1784189512.46 total 1501.48 last 48.96) - open/close 41917 (time 1784189522.51 total 1511.53 last 48.48) - open/close 42472 (time 1784189532.52 total 1521.54 last 55.45) - open/close 43065 (time 1784189542.53 total 1531.54 last 59.26) - open/close 43597 (time 1784189552.54 total 1541.56 last 53.12) - open/close 44174 (time 1784189562.55 total 1551.56 last 57.68) - open/close 44768 (time 1784189572.57 total 1561.59 last 59.27) - open/close 45328 (time 1784189582.57 total 1571.59 last 55.98) - open/close 45810 (time 1784189592.60 total 1581.62 last 48.07) - open/close 46278 (time 1784189602.60 total 1591.62 last 46.78) - open/close 46753 (time 1784189612.61 total 1601.62 last 47.49) - open/close 47295 (time 1784189622.62 total 1611.64 last 54.10) - open/close 47822 (time 1784189632.64 total 1621.66 last 52.61) - open/close 48375 (time 1784189642.65 total 1631.67 last 55.25) - open/close 48862 (time 1784189652.67 total 1641.69 last 48.60) - open/close 49303 (time 1784189662.68 total 1651.69 last 44.08) - open/close 49810 (time 1784189672.69 total 1661.71 last 50.63) total: 50000 open/close in 1665.06 seconds: 30.03 ops/second - unlinked 0 (time 1784189681 ; total 0 ; last 0) - unlinked 10000 (time 1784189895 ; total 214 ; last 214) - unlinked 20000 (time 1784190106 ; total 425 ; last 211) - unlinked 30000 (time 1784190303 ; total 622 ; last 197) - unlinked 40000 (time 1784190496 ; total 815 ; last 193) total: 50000 unlinks in 997 seconds: 50.150452 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg448-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg448-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg448-server stop mds service on oleg448-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg448-server unloading modules via unload_modules_local on: 'oleg448-server' oleg448-server: oleg448-server.virtnet: executing unload_modules_local oleg448-server: modules unloaded. pdsh@oleg448-client: oleg448-client: ssh exited with exit code 2 pdsh@oleg448-client: oleg448-server: ssh exited with exit code 2 pdsh@oleg448-client: oleg448-client: ssh exited with exit code 2 pdsh@oleg448-client: oleg448-server: ssh exited with exit code 2