== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 19:35:28 (1785454528) start mds service on oleg634-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg634-server' oleg634-server: oleg634-server.virtnet: executing load_modules_local oleg634-server: Loading modules from /home/green/git/lustre-release/lustre oleg634-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg634-server: detected 4 online CPUs by sysfs oleg634-server: MODOPTS_LIBCFS= oleg634-server: Force libcfs to create 2 CPU partitions oleg634-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg634-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg634-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg634-server: oleg634-server.virtnet: executing set_default_debug -1 all pdsh@oleg634-client: oleg634-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg634-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg634-server: oleg634-server.virtnet: executing set_default_debug -1 all pdsh@oleg634-client: oleg634-server: ssh exited with exit code 1 Started lustre-OST0000 oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg634-client.virtnet: -o user_xattr,flock 192.168.206.134@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 232 (time 1785454614.93 total 10.01 last 23.18) - open/close 471 (time 1785454625.64 total 20.71 last 22.34) - open/close 724 (time 1785454636.39 total 31.46 last 23.53) - open/close 964 (time 1785454646.39 total 41.47 last 23.98) - open/close 1198 (time 1785454657.33 total 52.40 last 21.40) - open/close 1467 (time 1785454668.33 total 63.40 last 24.45) - open/close 1734 (time 1785454678.33 total 73.40 last 26.69) - open/close 1963 (time 1785454688.93 total 84.00 last 21.60) - open/close 2103 (time 1785454699.90 total 94.97 last 12.77) - open/close 2331 (time 1785454709.90 total 104.97 last 22.80) - open/close 2581 (time 1785454720.70 total 115.77 last 23.15) - open/close 2806 (time 1785454731.10 total 126.18 last 21.63) - open/close 3063 (time 1785454741.34 total 136.41 last 25.11) - open/close 3331 (time 1785454751.35 total 146.42 last 26.78) - open/close 3523 (time 1785454761.36 total 156.43 last 19.17) - open/close 3717 (time 1785454772.13 total 167.21 last 18.01) - open/close 3946 (time 1785454782.37 total 177.44 last 22.37) - open/close 4148 (time 1785454792.39 total 187.46 last 20.18) - open/close 4285 (time 1785454806.44 total 201.51 last 9.75) - open/close 4519 (time 1785454816.46 total 211.54 last 23.34) - open/close 4756 (time 1785454827.10 total 222.17 last 22.28) - open/close 5005 (time 1785454837.11 total 232.18 last 24.89) - open/close 5194 (time 1785454848.12 total 243.19 last 17.16) - open/close 5428 (time 1785454858.56 total 253.63 last 22.42) - open/close 5558 (time 1785454868.56 total 263.64 last 12.99) - open/close 5741 (time 1785454879.39 total 274.46 last 16.90) - open/close 5866 (time 1785454889.40 total 284.47 last 12.49) - open/close 6108 (time 1785454899.75 total 294.82 last 23.38) - open/close 6395 (time 1785454910.71 total 305.78 last 26.18) - open/close 6558 (time 1785454920.80 total 315.87 last 16.16) - open/close 6763 (time 1785454932.98 total 328.05 last 16.83) - open/close 7039 (time 1785454942.99 total 338.07 last 27.56) - open/close 7255 (time 1785454953.01 total 348.08 last 21.57) - open/close 7522 (time 1785454963.75 total 358.82 last 24.86) - open/close 7799 (time 1785454974.64 total 369.71 last 25.44) - open/close 8000 (time 1785454984.64 total 379.71 last 20.10) - open/close 8245 (time 1785454994.91 total 389.98 last 23.85) - open/close 8468 (time 1785455004.92 total 399.99 last 22.28) - open/close 8730 (time 1785455015.40 total 410.47 last 24.99) - open/close 8962 (time 1785455025.63 total 420.70 last 22.68) - open/close 9245 (time 1785455035.64 total 430.71 last 28.28) - open/close 9575 (time 1785455046.34 total 441.41 last 30.83) - open/close 9779 (time 1785455056.34 total 451.42 last 20.40) - open/close 10000 (time 1785455065.60 total 460.67 last 23.89) - open/close 10225 (time 1785455075.60 total 470.68 last 22.48) - open/close 10417 (time 1785455085.61 total 480.69 last 19.18) - open/close 10620 (time 1785455096.10 total 491.17 last 19.36) - open/close 10839 (time 1785455106.10 total 501.17 last 21.89) - open/close 11071 (time 1785455116.11 total 511.18 last 23.18) - open/close 11200 (time 1785455131.94 total 527.01 last 8.15) - open/close 11478 (time 1785455141.94 total 537.01 last 27.79) - open/close 11702 (time 1785455151.96 total 547.03 last 22.36) - open/close 11888 (time 1785455162.09 total 557.16 last 18.36) - open/close 11982 (time 1785455172.38 total 567.45 last 9.13) - open/close 12239 (time 1785455182.95 total 578.02 last 24.31) - open/close 12459 (time 1785455193.57 total 588.64 last 20.72) - open/close 12696 (time 1785455204.50 total 599.57 last 21.69) - open/close 12955 (time 1785455214.51 total 609.58 last 25.87) - open/close 13148 (time 1785455225.57 total 620.64 last 17.45) - open/close 13409 (time 1785455235.57 total 630.64 last 26.10) - open/close 13616 (time 1785455247.22 total 642.29 last 17.76) - open/close 13743 (time 1785455257.66 total 652.73 last 12.17) - open/close 13924 (time 1785455267.68 total 662.75 last 18.06) - open/close 14050 (time 1785455277.75 total 672.83 last 12.51) - open/close 14270 (time 1785455288.04 total 683.12 last 21.38) - open/close 14446 (time 1785455298.97 total 694.04 last 16.10) - open/close 14694 (time 1785455308.97 total 704.05 last 24.80) - open/close 14889 (time 1785455318.98 total 714.05 last 19.48) - open/close 15129 (time 1785455329.39 total 724.46 last 23.06) - open/close 15373 (time 1785455339.39 total 734.46 last 24.40) - open/close 15603 (time 1785455350.44 total 745.51 last 20.82) - open/close 15791 (time 1785455361.01 total 756.08 last 17.78) - open/close 16003 (time 1785455371.21 total 766.29 last 20.78) - open/close 16272 (time 1785455381.22 total 776.29 last 26.90) - open/close 16464 (time 1785455391.24 total 786.31 last 19.16) - open/close 16699 (time 1785455401.28 total 796.35 last 23.40) - open/close 16905 (time 1785455412.07 total 807.14 last 19.09) - open/close 17138 (time 1785455422.11 total 817.18 last 23.20) - open/close 17374 (time 1785455432.93 total 828.00 last 21.82) - open/close 17499 (time 1785455442.95 total 838.03 last 12.47) - open/close 17749 (time 1785455453.92 total 848.99 last 22.79) - open/close 17899 (time 1785455464.10 total 859.17 last 14.74) - open/close 18041 (time 1785455475.02 total 870.09 last 13.00) - open/close 18212 (time 1785455485.95 total 881.02 last 15.65) - open/close 18407 (time 1785455496.89 total 891.97 last 17.82) - open/close 18668 (time 1785455507.86 total 902.94 last 23.79) - open/close 18940 (time 1785455518.82 total 913.89 last 24.84) - open/close 19198 (time 1785455528.83 total 923.90 last 25.76) - open/close 19902 (time 1785455538.85 total 933.92 last 70.27) - open/close 20593 (time 1785455548.86 total 943.93 last 69.05) - open/close 21258 (time 1785455558.87 total 953.94 last 66.42) - open/close 21828 (time 1785455568.88 total 963.95 last 56.94) - open/close 22441 (time 1785455578.89 total 973.96 last 61.24) - open/close 23136 (time 1785455588.89 total 983.96 last 69.49) - open/close 23780 (time 1785455598.90 total 993.97 last 64.36) - open/close 24416 (time 1785455608.92 total 1003.99 last 63.49) - open/close 25112 (time 1785455618.92 total 1013.99 last 69.55) - open/close 25819 (time 1785455628.93 total 1024.00 last 70.64) - open/close 26546 (time 1785455638.95 total 1034.03 last 72.54) - open/close 27260 (time 1785455648.96 total 1044.03 last 71.35) - open/close 27955 (time 1785455658.98 total 1054.05 last 69.35) - open/close 28576 (time 1785455668.99 total 1064.06 last 62.06) - open/close 29244 (time 1785455679.01 total 1074.09 last 66.64) - open/close 29984 (time 1785455689.03 total 1084.10 last 73.87) - open/close 30770 (time 1785455699.04 total 1094.11 last 78.57) - open/close 31508 (time 1785455709.05 total 1104.12 last 73.69) - open/close 32158 (time 1785455719.07 total 1114.14 last 64.87) - open/close 32851 (time 1785455729.08 total 1124.15 last 69.23) - open/close 33604 (time 1785455739.08 total 1134.15 last 75.29) - open/close 34326 (time 1785455749.09 total 1144.16 last 72.17) - open/close 35031 (time 1785455759.09 total 1154.16 last 70.46) - open/close 35767 (time 1785455769.09 total 1164.16 last 73.60) - open/close 36486 (time 1785455779.10 total 1174.18 last 71.81) - open/close 37158 (time 1785455789.11 total 1184.18 last 67.18) - open/close 37864 (time 1785455799.11 total 1194.18 last 70.57) - open/close 38562 (time 1785455809.12 total 1204.19 last 69.72) - open/close 39215 (time 1785455819.14 total 1214.21 last 65.21) - open/close 39871 (time 1785455829.14 total 1224.21 last 65.60) - open/close 40568 (time 1785455839.14 total 1234.21 last 69.67) - open/close 41255 (time 1785455849.15 total 1244.22 last 68.66) - open/close 41974 (time 1785455859.16 total 1254.23 last 71.81) - open/close 42668 (time 1785455869.16 total 1264.23 last 69.40) - open/close 43400 (time 1785455879.17 total 1274.24 last 73.16) - open/close 43985 (time 1785455889.17 total 1284.24 last 58.48) - open/close 44697 (time 1785455899.17 total 1294.24 last 71.18) - open/close 45364 (time 1785455909.19 total 1304.26 last 66.62) - open/close 46021 (time 1785455919.19 total 1314.26 last 65.69) - open/close 46667 (time 1785455929.22 total 1324.29 last 64.41) - open/close 47351 (time 1785455939.22 total 1334.29 last 68.39) - open/close 48063 (time 1785455949.23 total 1344.30 last 71.11) - open/close 48755 (time 1785455959.24 total 1354.31 last 69.13) - open/close 49462 (time 1785455969.25 total 1364.32 last 70.67) total: 50000 open/close in 1373.17 seconds: 36.41 ops/second - unlinked 0 (time 1785455982 ; total 0 ; last 0) - unlinked 10000 (time 1785456135 ; total 153 ; last 153) - unlinked 20000 (time 1785456283 ; total 301 ; last 148) - unlinked 30000 (time 1785456436 ; total 454 ; last 153) - unlinked 40000 (time 1785456577 ; total 595 ; last 141) total: 50000 unlinks in 751 seconds: 66.577896 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg634-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg634-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg634-server stop mds service on oleg634-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg634-server unloading modules via unload_modules_local on: 'oleg634-server' oleg634-server: oleg634-server.virtnet: executing unload_modules_local oleg634-server: modules unloaded. pdsh@oleg634-client: oleg634-client: ssh exited with exit code 2 pdsh@oleg634-client: oleg634-server: ssh exited with exit code 2 pdsh@oleg634-client: oleg634-client: ssh exited with exit code 2 pdsh@oleg634-client: oleg634-server: ssh exited with exit code 2