== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 07:06:55 (1787828815) start mds service on oleg646-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg646-server' oleg646-server: oleg646-server.virtnet: executing load_modules_local oleg646-server: Loading modules from /home/green/git/lustre-release/lustre oleg646-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg646-server: detected 4 online CPUs by sysfs oleg646-server: MODOPTS_LIBCFS= oleg646-server: Force libcfs to create 2 CPU partitions oleg646-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg646-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg646-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg646-server: oleg646-server.virtnet: executing set_default_debug -1 all pdsh@oleg646-client: oleg646-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg646-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg646-server: oleg646-server.virtnet: executing set_default_debug -1 all pdsh@oleg646-client: oleg646-server: ssh exited with exit code 1 Started lustre-OST0000 oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg646-client.virtnet: -o user_xattr,flock 192.168.206.146@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 196 (time 1787828900.20 total 10.01 last 19.58) - open/close 465 (time 1787828911.02 total 20.83 last 24.86) - open/close 730 (time 1787828921.83 total 31.65 last 24.51) - open/close 978 (time 1787828932.57 total 42.38 last 23.10) - open/close 1189 (time 1787828942.96 total 52.78 last 20.31) - open/close 1448 (time 1787828954.02 total 63.83 last 23.43) - open/close 1655 (time 1787828964.03 total 73.84 last 20.67) - open/close 1932 (time 1787828974.31 total 84.12 last 26.95) - open/close 2110 (time 1787828984.32 total 94.13 last 17.77) - open/close 2294 (time 1787828994.97 total 104.78 last 17.28) - open/close 2493 (time 1787829005.87 total 115.69 last 18.25) - open/close 2752 (time 1787829015.89 total 125.70 last 25.86) - open/close 2815 (time 1787829026.67 total 136.49 last 5.84) - open/close 2999 (time 1787829036.77 total 146.58 last 18.22) - open/close 3134 (time 1787829047.08 total 156.89 last 13.10) - open/close 3250 (time 1787829057.08 total 166.89 last 11.60) - open/close 3462 (time 1787829067.24 total 177.05 last 20.88) - open/close 3755 (time 1787829077.57 total 187.38 last 28.36) - open/close 3989 (time 1787829090.28 total 200.09 last 18.41) - open/close 4221 (time 1787829101.21 total 211.02 last 21.23) - open/close 4356 (time 1787829112.19 total 222.00 last 12.29) - open/close 4458 (time 1787829122.79 total 232.61 last 9.62) - open/close 4643 (time 1787829132.96 total 242.77 last 18.20) - open/close 4843 (time 1787829143.27 total 253.08 last 19.39) - open/close 5012 (time 1787829153.28 total 263.09 last 16.89) - open/close 5175 (time 1787829163.38 total 273.19 last 16.14) - open/close 5401 (time 1787829173.86 total 283.67 last 21.57) - open/close 5582 (time 1787829184.42 total 294.24 last 17.13) - open/close 5747 (time 1787829194.44 total 304.26 last 16.47) - open/close 5845 (time 1787829204.77 total 314.58 last 9.49) - open/close 6005 (time 1787829215.26 total 325.08 last 15.25) - open/close 6170 (time 1787829225.64 total 335.46 last 15.89) - open/close 6348 (time 1787829236.74 total 346.55 last 16.05) - open/close 6625 (time 1787829247.79 total 357.60 last 25.06) - open/close 6791 (time 1787829259.18 total 368.99 last 14.57) - open/close 6992 (time 1787829269.19 total 379.00 last 20.09) - open/close 7239 (time 1787829279.99 total 389.81 last 22.86) - open/close 7442 (time 1787829290.00 total 399.81 last 20.29) - open/close 7626 (time 1787829301.67 total 411.48 last 15.77) - open/close 7883 (time 1787829311.69 total 421.51 last 25.64) - open/close 8124 (time 1787829322.67 total 432.48 last 21.96) - open/close 8305 (time 1787829332.67 total 442.48 last 18.10) - open/close 8378 (time 1787829342.67 total 452.49 last 7.30) - open/close 8623 (time 1787829353.64 total 463.45 last 22.34) - open/close 8829 (time 1787829363.65 total 473.46 last 20.57) - open/close 9085 (time 1787829373.66 total 483.47 last 25.58) - open/close 9316 (time 1787829383.67 total 493.49 last 23.07) - open/close 9566 (time 1787829394.42 total 504.23 last 23.27) - open/close 9678 (time 1787829404.52 total 514.33 last 11.09) - open/close 9876 (time 1787829414.53 total 524.34 last 19.78) - open/close 10000 (time 1787829420.04 total 529.85 last 22.52) - open/close 10242 (time 1787829430.77 total 540.58 last 22.55) - open/close 10395 (time 1787829440.78 total 550.60 last 15.27) - open/close 10597 (time 1787829451.36 total 561.18 last 19.10) - open/close 10853 (time 1787829461.37 total 571.19 last 25.57) - open/close 11023 (time 1787829471.93 total 581.74 last 16.11) - open/close 11134 (time 1787829481.95 total 591.77 last 11.07) - open/close 11300 (time 1787829491.97 total 601.79 last 16.57) - open/close 11426 (time 1787829501.98 total 611.80 last 12.59) - open/close 11546 (time 1787829511.99 total 621.80 last 12.00) - open/close 11693 (time 1787829522.28 total 632.10 last 14.28) - open/close 11957 (time 1787829534.88 total 644.69 last 20.96) - open/close 12175 (time 1787829544.88 total 654.70 last 21.79) - open/close 12313 (time 1787829555.38 total 665.19 last 13.15) - open/close 12513 (time 1787829565.87 total 675.69 last 19.06) - open/close 12685 (time 1787829575.89 total 685.70 last 17.18) - open/close 12880 (time 1787829585.89 total 695.70 last 19.50) - open/close 13113 (time 1787829596.39 total 706.20 last 22.20) - open/close 13273 (time 1787829608.95 total 718.76 last 12.74) - open/close 13344 (time 1787829620.20 total 730.01 last 6.31) - open/close 13447 (time 1787829630.38 total 740.20 last 10.11) - open/close 13608 (time 1787829640.39 total 750.21 last 16.09) - open/close 13729 (time 1787829652.37 total 762.18 last 10.10) - open/close 13951 (time 1787829662.40 total 772.21 last 22.15) - open/close 14129 (time 1787829672.41 total 782.22 last 17.78) - open/close 14317 (time 1787829682.42 total 792.24 last 18.77) - open/close 14539 (time 1787829692.43 total 802.25 last 22.18) - open/close 14742 (time 1787829704.43 total 814.25 last 16.92) - open/close 14842 (time 1787829714.67 total 824.49 last 9.77) - open/close 15029 (time 1787829725.22 total 835.04 last 17.73) - open/close 15096 (time 1787829735.24 total 845.05 last 6.69) - open/close 15381 (time 1787829746.24 total 856.05 last 25.91) - open/close 15492 (time 1787829756.24 total 866.05 last 11.10) - open/close 15631 (time 1787829767.21 total 877.02 last 12.67) - open/close 15782 (time 1787829777.56 total 887.37 last 14.59) - open/close 15899 (time 1787829787.56 total 897.38 last 11.69) - open/close 16094 (time 1787829797.69 total 907.50 last 19.26) - open/close 16324 (time 1787829808.37 total 918.19 last 21.52) - open/close 16575 (time 1787829819.12 total 928.93 last 23.36) - open/close 16742 (time 1787829829.14 total 938.96 last 16.66) - open/close 16937 (time 1787829839.15 total 948.97 last 19.48) - open/close 17073 (time 1787829849.37 total 959.18 last 13.31) - open/close 17348 (time 1787829859.96 total 969.78 last 25.97) - open/close 17519 (time 1787829875.65 total 985.46 last 10.90) - open/close 17746 (time 1787829885.65 total 995.47 last 22.68) - open/close 18086 (time 1787829895.67 total 1005.48 last 33.95) - open/close 18751 (time 1787829905.67 total 1015.49 last 66.46) - open/close 19431 (time 1787829915.68 total 1025.49 last 67.96) - open/close 20000 (time 1787829923.36 total 1033.17 last 74.13) - open/close 20700 (time 1787829933.36 total 1043.18 last 69.94) - open/close 21448 (time 1787829943.37 total 1053.19 last 74.73) - open/close 22223 (time 1787829953.38 total 1063.19 last 77.48) - open/close 22897 (time 1787829963.38 total 1073.19 last 67.38) - open/close 23552 (time 1787829973.39 total 1083.20 last 65.46) - open/close 24256 (time 1787829983.40 total 1093.21 last 70.32) - open/close 24985 (time 1787829993.40 total 1103.21 last 72.90) - open/close 25760 (time 1787830003.40 total 1113.22 last 77.46) - open/close 26431 (time 1787830013.41 total 1123.23 last 67.03) - open/close 27090 (time 1787830023.43 total 1133.24 last 65.79) - open/close 27805 (time 1787830033.44 total 1143.25 last 71.45) - open/close 28541 (time 1787830043.44 total 1153.25 last 73.59) - open/close 29274 (time 1787830053.45 total 1163.26 last 73.24) - open/close 29940 (time 1787830063.45 total 1173.27 last 66.56) - open/close 30625 (time 1787830073.46 total 1183.27 last 68.46) - open/close 31338 (time 1787830083.47 total 1193.29 last 71.22) - open/close 32055 (time 1787830093.48 total 1203.29 last 71.65) - open/close 32776 (time 1787830103.49 total 1213.30 last 72.03) - open/close 33491 (time 1787830113.49 total 1223.31 last 71.47) - open/close 34160 (time 1787830123.50 total 1233.32 last 66.83) - open/close 34931 (time 1787830133.50 total 1243.32 last 77.08) - open/close 35526 (time 1787830143.51 total 1253.33 last 59.45) - open/close 36154 (time 1787830153.55 total 1263.36 last 62.60) - open/close 36740 (time 1787830163.56 total 1273.37 last 58.53) - open/close 37423 (time 1787830173.57 total 1283.39 last 68.20) - open/close 37964 (time 1787830183.59 total 1293.40 last 54.01) - open/close 38457 (time 1787830193.59 total 1303.41 last 49.28) - open/close 38907 (time 1787830203.66 total 1313.47 last 44.71) - open/close 39431 (time 1787830213.69 total 1323.50 last 52.26) - open/close 40000 (time 1787830222.98 total 1332.79 last 61.25) - open/close 40582 (time 1787830232.98 total 1342.79 last 58.18) - open/close 41208 (time 1787830242.98 total 1352.79 last 62.59) - open/close 41911 (time 1787830252.99 total 1362.81 last 70.20) - open/close 42576 (time 1787830262.99 total 1372.81 last 66.50) - open/close 43251 (time 1787830273.00 total 1382.81 last 67.48) - open/close 43960 (time 1787830283.00 total 1392.82 last 70.87) - open/close 44650 (time 1787830293.01 total 1402.82 last 68.97) - open/close 45282 (time 1787830303.01 total 1412.83 last 63.16) - open/close 45908 (time 1787830313.04 total 1422.85 last 62.45) - open/close 46568 (time 1787830323.04 total 1432.86 last 65.96) - open/close 47302 (time 1787830333.06 total 1442.87 last 73.29) - open/close 48005 (time 1787830343.06 total 1452.88 last 70.26) - open/close 48657 (time 1787830353.06 total 1462.88 last 65.20) - open/close 49266 (time 1787830363.07 total 1472.88 last 60.86) - open/close 49903 (time 1787830373.08 total 1482.90 last 63.62) total: 50000 open/close in 1484.31 seconds: 33.69 ops/second - unlinked 0 (time 1787830379 ; total 0 ; last 0) - unlinked 10000 (time 1787830540 ; total 161 ; last 161) - unlinked 20000 (time 1787830692 ; total 313 ; last 152) - unlinked 30000 (time 1787830845 ; total 466 ; last 153) - unlinked 40000 (time 1787830990 ; total 611 ; last 145) total: 50000 unlinks in 759 seconds: 65.876152 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg646-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg646-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg646-server stop mds service on oleg646-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg646-server unloading modules via unload_modules_local on: 'oleg646-server' oleg646-server: oleg646-server.virtnet: executing unload_modules_local oleg646-server: modules unloaded. pdsh@oleg646-client: oleg646-client: ssh exited with exit code 2 pdsh@oleg646-client: oleg646-server: ssh exited with exit code 2 pdsh@oleg646-client: oleg646-client: ssh exited with exit code 2 pdsh@oleg646-client: oleg646-server: ssh exited with exit code 2