== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:11:11 (1787616671) start mds service on oleg358-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg358-server' oleg358-server: oleg358-server.virtnet: executing load_modules_local oleg358-server: Loading modules from /home/green/git/lustre-release/lustre oleg358-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg358-server: detected 4 online CPUs by sysfs oleg358-server: MODOPTS_LIBCFS= oleg358-server: Force libcfs to create 2 CPU partitions oleg358-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg358-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg358-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg358-server: oleg358-server.virtnet: executing set_default_debug -1 all pdsh@oleg358-client: oleg358-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg358-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg358-server: oleg358-server.virtnet: executing set_default_debug -1 all pdsh@oleg358-client: oleg358-server: ssh exited with exit code 1 Started lustre-OST0000 oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg358-client.virtnet: -o user_xattr,flock 192.168.203.158@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 128 (time 1787616756.61 total 12.62 last 10.15) - open/close 250 (time 1787616768.50 total 24.51 last 10.26) - open/close 403 (time 1787616778.81 total 34.82 last 14.84) - open/close 628 (time 1787616788.83 total 44.84 last 22.46) - open/close 880 (time 1787616799.89 total 55.90 last 22.77) - open/close 1130 (time 1787616810.94 total 66.95 last 22.64) - open/close 1294 (time 1787616821.35 total 77.36 last 15.74) - open/close 1435 (time 1787616831.47 total 87.48 last 13.94) - open/close 1618 (time 1787616842.16 total 98.17 last 17.12) - open/close 1784 (time 1787616852.41 total 108.42 last 16.20) - open/close 2010 (time 1787616862.41 total 118.42 last 22.59) - open/close 2214 (time 1787616872.42 total 128.43 last 20.39) - open/close 2352 (time 1787616882.43 total 138.44 last 13.78) - open/close 2537 (time 1787616893.29 total 149.30 last 17.03) - open/close 2774 (time 1787616903.30 total 159.31 last 23.68) - open/close 3005 (time 1787616914.35 total 170.36 last 20.92) - open/close 3213 (time 1787616924.67 total 180.68 last 20.15) - open/close 3384 (time 1787616935.64 total 191.65 last 15.60) - open/close 3519 (time 1787616945.64 total 201.65 last 13.50) - open/close 3693 (time 1787616956.59 total 212.60 last 15.89) - open/close 3836 (time 1787616967.18 total 223.19 last 13.50) - open/close 3992 (time 1787616977.18 total 233.19 last 15.60) - open/close 4150 (time 1787616987.19 total 243.20 last 15.79) - open/close 4267 (time 1787616997.29 total 253.30 last 11.58) - open/close 4334 (time 1787617007.88 total 263.89 last 6.33) - open/close 4496 (time 1787617017.88 total 273.89 last 16.20) - open/close 4602 (time 1787617028.91 total 284.92 last 9.61) - open/close 4711 (time 1787617039.80 total 295.81 last 10.01) - open/close 4895 (time 1787617049.81 total 305.82 last 18.39) - open/close 5101 (time 1787617059.82 total 315.83 last 20.57) - open/close 5342 (time 1787617069.84 total 325.85 last 24.05) - open/close 5513 (time 1787617080.65 total 336.66 last 15.82) - open/close 5728 (time 1787617090.66 total 346.67 last 21.47) - open/close 5947 (time 1787617101.17 total 357.18 last 20.85) - open/close 6178 (time 1787617111.18 total 367.19 last 23.08) - open/close 6315 (time 1787617121.32 total 377.33 last 13.50) - open/close 6455 (time 1787617131.64 total 387.66 last 13.56) - open/close 6643 (time 1787617141.65 total 397.66 last 18.79) - open/close 6838 (time 1787617152.31 total 408.32 last 18.30) - open/close 7058 (time 1787617162.32 total 418.33 last 21.97) - open/close 7275 (time 1787617172.46 total 428.47 last 21.39) - open/close 7506 (time 1787617182.49 total 438.50 last 23.04) - open/close 7698 (time 1787617193.01 total 449.02 last 18.25) - open/close 7783 (time 1787617204.68 total 460.69 last 7.28) - open/close 7883 (time 1787617215.10 total 471.11 last 9.60) - open/close 8128 (time 1787617225.10 total 481.11 last 24.50) - open/close 8344 (time 1787617235.11 total 491.12 last 21.58) - open/close 8564 (time 1787617245.11 total 501.12 last 21.99) - open/close 8720 (time 1787617255.49 total 511.50 last 15.03) - open/close 8891 (time 1787617265.51 total 521.52 last 17.07) - open/close 9005 (time 1787617275.53 total 531.54 last 11.38) - open/close 9158 (time 1787617286.08 total 542.09 last 14.49) - open/close 9412 (time 1787617296.57 total 552.58 last 24.23) - open/close 9643 (time 1787617306.57 total 562.58 last 23.09) - open/close 9823 (time 1787617316.74 total 572.75 last 17.71) - open/close 9958 (time 1787617327.42 total 583.43 last 12.64) - open/close 10140 (time 1787617337.43 total 593.44 last 18.18) - open/close 10243 (time 1787617347.45 total 603.46 last 10.28) - open/close 10409 (time 1787617357.53 total 613.54 last 16.47) - open/close 10516 (time 1787617367.53 total 623.54 last 10.70) - open/close 10635 (time 1787617377.54 total 633.55 last 11.89) - open/close 10772 (time 1787617387.59 total 643.60 last 13.63) - open/close 10934 (time 1787617398.51 total 654.52 last 14.84) - open/close 11128 (time 1787617411.45 total 667.46 last 14.99) - open/close 11347 (time 1787617422.49 total 678.50 last 19.84) - open/close 11511 (time 1787617432.49 total 688.50 last 16.39) - open/close 11656 (time 1787617442.59 total 698.60 last 14.35) - open/close 11729 (time 1787617452.92 total 708.93 last 7.07) - open/close 11817 (time 1787617463.23 total 719.24 last 8.54) - open/close 12023 (time 1787617473.26 total 729.27 last 20.52) - open/close 12252 (time 1787617483.27 total 739.28 last 22.90) - open/close 12466 (time 1787617493.27 total 749.28 last 21.40) - open/close 12687 (time 1787617503.27 total 759.28 last 22.10) - open/close 12907 (time 1787617513.80 total 769.81 last 20.90) - open/close 13078 (time 1787617523.80 total 779.81 last 17.09) - open/close 13319 (time 1787617534.15 total 790.16 last 23.30) - open/close 13507 (time 1787617545.14 total 801.15 last 17.10) - open/close 13722 (time 1787617555.16 total 811.17 last 21.46) - open/close 13780 (time 1787617565.18 total 821.19 last 5.79) - open/close 13977 (time 1787617575.19 total 831.20 last 19.68) - open/close 14095 (time 1787617586.16 total 842.18 last 10.75) - open/close 14211 (time 1787617596.18 total 852.19 last 11.59) - open/close 14383 (time 1787617606.19 total 862.20 last 17.18) - open/close 14633 (time 1787617616.19 total 872.20 last 25.00) - open/close 14878 (time 1787617626.20 total 882.21 last 24.48) - open/close 15079 (time 1787617636.53 total 892.54 last 19.46) - open/close 15343 (time 1787617647.37 total 903.38 last 24.36) - open/close 15577 (time 1787617658.10 total 914.11 last 21.79) - open/close 15792 (time 1787617668.41 total 924.42 last 20.86) - open/close 15989 (time 1787617678.70 total 934.71 last 19.14) - open/close 16139 (time 1787617689.66 total 945.67 last 13.70) - open/close 16381 (time 1787617699.68 total 955.69 last 24.15) - open/close 16600 (time 1787617709.71 total 965.72 last 21.83) - open/close 16802 (time 1787617720.62 total 976.63 last 18.51) - open/close 17023 (time 1787617731.38 total 987.39 last 20.53) - open/close 17292 (time 1787617742.14 total 998.15 last 25.01) - open/close 17565 (time 1787617752.14 total 1008.15 last 27.29) - open/close 18253 (time 1787617762.16 total 1018.17 last 68.70) - open/close 18916 (time 1787617772.16 total 1028.17 last 66.25) - open/close 19594 (time 1787617782.17 total 1038.18 last 67.74) - open/close 20000 (time 1787617788.11 total 1044.12 last 68.31) - open/close 20696 (time 1787617798.14 total 1054.15 last 69.45) - open/close 21374 (time 1787617808.14 total 1064.15 last 67.79) - open/close 22018 (time 1787617818.14 total 1074.15 last 64.35) - open/close 22698 (time 1787617828.15 total 1084.16 last 67.98) - open/close 23297 (time 1787617838.17 total 1094.18 last 59.79) - open/close 24011 (time 1787617848.17 total 1104.18 last 71.39) - open/close 24646 (time 1787617858.17 total 1114.18 last 63.46) - open/close 25215 (time 1787617868.18 total 1124.19 last 56.88) - open/close 25807 (time 1787617878.18 total 1134.19 last 59.18) - open/close 26431 (time 1787617888.19 total 1144.20 last 62.31) - open/close 27054 (time 1787617898.21 total 1154.22 last 62.23) - open/close 27706 (time 1787617908.23 total 1164.24 last 65.05) - open/close 28367 (time 1787617918.25 total 1174.26 last 65.98) - open/close 29041 (time 1787617928.25 total 1184.26 last 67.40) - open/close 29743 (time 1787617938.26 total 1194.27 last 70.09) - open/close 30000 (time 1787617941.81 total 1197.82 last 72.39) - open/close 30693 (time 1787617951.82 total 1207.83 last 69.22) - open/close 31396 (time 1787617961.83 total 1217.84 last 70.26) - open/close 32055 (time 1787617971.83 total 1227.84 last 65.88) - open/close 32699 (time 1787617981.84 total 1237.85 last 64.36) - open/close 33371 (time 1787617991.85 total 1247.86 last 67.10) - open/close 34030 (time 1787618001.87 total 1257.88 last 65.83) - open/close 34632 (time 1787618011.88 total 1267.89 last 60.13) - open/close 35326 (time 1787618021.88 total 1277.89 last 69.36) - open/close 36021 (time 1787618031.91 total 1287.92 last 69.34) - open/close 36697 (time 1787618041.92 total 1297.93 last 67.54) - open/close 37380 (time 1787618051.92 total 1307.93 last 68.27) - open/close 38008 (time 1787618061.92 total 1317.94 last 62.77) - open/close 38646 (time 1787618071.94 total 1327.95 last 63.73) - open/close 39251 (time 1787618081.94 total 1337.95 last 60.46) - open/close 39884 (time 1787618091.95 total 1347.96 last 63.27) - open/close 40530 (time 1787618101.95 total 1357.96 last 64.59) - open/close 41153 (time 1787618111.96 total 1367.97 last 62.25) - open/close 41836 (time 1787618121.96 total 1377.97 last 68.28) - open/close 42531 (time 1787618131.97 total 1387.98 last 69.41) - open/close 43300 (time 1787618141.98 total 1397.99 last 76.81) - open/close 43995 (time 1787618151.99 total 1408.00 last 69.48) - open/close 44698 (time 1787618162.00 total 1418.01 last 70.24) - open/close 45361 (time 1787618172.00 total 1428.01 last 66.26) - open/close 46096 (time 1787618182.01 total 1438.02 last 73.46) - open/close 46828 (time 1787618192.01 total 1448.02 last 73.20) - open/close 47513 (time 1787618202.01 total 1458.02 last 68.47) - open/close 48186 (time 1787618212.02 total 1468.03 last 67.27) - open/close 48789 (time 1787618222.04 total 1478.05 last 60.17) - open/close 49474 (time 1787618232.05 total 1488.06 last 68.44) total: 50000 open/close in 1495.30 seconds: 33.44 ops/second - unlinked 0 (time 1787618242 ; total 0 ; last 0) - unlinked 10000 (time 1787618399 ; total 157 ; last 157) - unlinked 20000 (time 1787618553 ; total 311 ; last 154) - unlinked 30000 (time 1787618713 ; total 471 ; last 160) - unlinked 40000 (time 1787618874 ; total 632 ; last 161) total: 50000 unlinks in 800 seconds: 62.500000 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg358-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg358-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg358-server stop mds service on oleg358-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg358-server unloading modules via unload_modules_local on: 'oleg358-server' oleg358-server: oleg358-server.virtnet: executing unload_modules_local oleg358-server: modules unloaded. pdsh@oleg358-client: oleg358-client: ssh exited with exit code 2 pdsh@oleg358-client: oleg358-server: ssh exited with exit code 2 pdsh@oleg358-client: oleg358-client: ssh exited with exit code 2 pdsh@oleg358-client: oleg358-server: ssh exited with exit code 2