== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:47:22 (1787863642) start mds service on oleg139-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg139-server' oleg139-server: oleg139-server.virtnet: executing load_modules_local oleg139-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg139-server: Loading modules from /home/green/git/lustre-release/lustre oleg139-server: detected 4 online CPUs by sysfs oleg139-server: MODOPTS_LIBCFS= oleg139-server: Force libcfs to create 2 CPU partitions oleg139-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg139-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg139-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg139-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-OST0000 oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg139-client.virtnet: -o user_xattr,flock 192.168.201.139@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 184 (time 1787863729.82 total 10.02 last 18.36) - open/close 439 (time 1787863739.82 total 20.03 last 25.49) - open/close 647 (time 1787863749.84 total 30.04 last 20.77) - open/close 809 (time 1787863760.31 total 40.51 last 15.47) - open/close 933 (time 1787863770.41 total 50.61 last 12.27) - open/close 1160 (time 1787863780.71 total 60.91 last 22.05) - open/close 1297 (time 1787863790.92 total 71.12 last 13.41) - open/close 1428 (time 1787863800.94 total 81.14 last 13.08) - open/close 1593 (time 1787863811.40 total 91.60 last 15.78) - open/close 1825 (time 1787863821.42 total 101.62 last 23.15) - open/close 1963 (time 1787863834.16 total 114.37 last 10.83) - open/close 2214 (time 1787863844.17 total 124.37 last 25.08) - open/close 2346 (time 1787863854.19 total 134.39 last 13.17) - open/close 2455 (time 1787863864.22 total 144.43 last 10.86) - open/close 2670 (time 1787863874.23 total 154.43 last 21.49) - open/close 2824 (time 1787863885.18 total 165.38 last 14.07) - open/close 3064 (time 1787863895.79 total 175.99 last 22.62) - open/close 3338 (time 1787863906.40 total 186.60 last 25.83) - open/close 3557 (time 1787863916.40 total 196.60 last 21.89) - open/close 3789 (time 1787863927.07 total 207.27 last 21.75) - open/close 4063 (time 1787863937.08 total 217.28 last 27.38) - open/close 4269 (time 1787863947.30 total 227.50 last 20.15) - open/close 4452 (time 1787863957.41 total 237.61 last 18.09) - open/close 4690 (time 1787863970.43 total 250.63 last 18.29) - open/close 4869 (time 1787863980.54 total 260.74 last 17.70) - open/close 5087 (time 1787863990.55 total 270.75 last 21.77) - open/close 5299 (time 1787864000.56 total 280.76 last 21.19) - open/close 5440 (time 1787864010.80 total 291.00 last 13.76) - open/close 5543 (time 1787864021.81 total 302.01 last 9.36) - open/close 5757 (time 1787864032.45 total 312.66 last 20.10) - open/close 5969 (time 1787864042.91 total 323.11 last 20.27) - open/close 6215 (time 1787864053.80 total 334.00 last 22.60) - open/close 6362 (time 1787864063.83 total 344.04 last 14.64) - open/close 6586 (time 1787864073.84 total 354.04 last 22.39) - open/close 6793 (time 1787864083.86 total 364.06 last 20.65) - open/close 7058 (time 1787864093.86 total 374.06 last 26.50) - open/close 7230 (time 1787864103.87 total 384.07 last 17.18) - open/close 7417 (time 1787864113.88 total 394.08 last 18.69) - open/close 7651 (time 1787864124.30 total 404.50 last 22.46) - open/close 7915 (time 1787864135.00 total 415.20 last 24.67) - open/close 8181 (time 1787864145.99 total 426.19 last 24.21) - open/close 8390 (time 1787864156.00 total 436.20 last 20.87) - open/close 8582 (time 1787864166.03 total 446.23 last 19.15) - open/close 8810 (time 1787864176.03 total 456.23 last 22.79) - open/close 8969 (time 1787864189.04 total 469.24 last 12.23) - open/close 9269 (time 1787864199.04 total 479.24 last 29.99) - open/close 9491 (time 1787864209.05 total 489.25 last 22.18) - open/close 9716 (time 1787864219.91 total 500.11 last 20.71) - open/close 9863 (time 1787864230.12 total 510.32 last 14.40) - open/close 10000 (time 1787864234.74 total 514.94 last 29.63) - open/close 10166 (time 1787864245.34 total 525.54 last 15.66) - open/close 10244 (time 1787864255.77 total 535.97 last 7.48) - open/close 10403 (time 1787864265.87 total 546.08 last 15.73) - open/close 10583 (time 1787864276.23 total 556.43 last 17.38) - open/close 10741 (time 1787864286.24 total 566.44 last 15.79) - open/close 10872 (time 1787864296.25 total 576.46 last 13.08) - open/close 10966 (time 1787864306.94 total 587.15 last 8.79) - open/close 11254 (time 1787864317.85 total 598.05 last 26.41) - open/close 11383 (time 1787864328.65 total 608.85 last 11.95) - open/close 11496 (time 1787864338.66 total 618.87 last 11.28) - open/close 11651 (time 1787864348.68 total 628.88 last 15.47) - open/close 11840 (time 1787864358.82 total 639.03 last 18.63) - open/close 12071 (time 1787864369.02 total 649.23 last 22.65) - open/close 12210 (time 1787864379.82 total 660.02 last 12.88) - open/close 12416 (time 1787864391.14 total 671.34 last 18.19) - open/close 12721 (time 1787864401.55 total 681.75 last 29.31) - open/close 13006 (time 1787864411.81 total 692.02 last 27.76) - open/close 13272 (time 1787864421.82 total 702.03 last 26.57) - open/close 13455 (time 1787864431.83 total 712.04 last 18.28) - open/close 13693 (time 1787864441.84 total 722.04 last 23.79) - open/close 13956 (time 1787864451.84 total 732.05 last 26.28) - open/close 14196 (time 1787864462.49 total 742.70 last 22.53) - open/close 14362 (time 1787864472.49 total 752.70 last 16.60) - open/close 14558 (time 1787864482.78 total 762.98 last 19.06) - open/close 14742 (time 1787864493.21 total 773.42 last 17.63) - open/close 15009 (time 1787864503.23 total 783.43 last 26.66) - open/close 15128 (time 1787864514.29 total 794.49 last 10.76) - open/close 15317 (time 1787864524.89 total 805.10 last 17.82) - open/close 15563 (time 1787864535.15 total 815.35 last 23.98) - open/close 15775 (time 1787864545.82 total 826.03 last 19.86) - open/close 15995 (time 1787864555.84 total 836.04 last 21.97) - open/close 16201 (time 1787864566.63 total 846.84 last 19.08) - open/close 16389 (time 1787864576.66 total 856.86 last 18.76) - open/close 16504 (time 1787864586.66 total 866.86 last 11.50) - open/close 16715 (time 1787864597.25 total 877.45 last 19.93) - open/close 16965 (time 1787864607.26 total 887.46 last 24.98) - open/close 17288 (time 1787864618.14 total 898.35 last 29.67) - open/close 17554 (time 1787864629.23 total 909.43 last 24.00) - open/close 17806 (time 1787864639.25 total 919.46 last 25.13) - open/close 18018 (time 1787864650.67 total 930.87 last 18.58) - open/close 18234 (time 1787864661.16 total 941.37 last 20.57) - open/close 18463 (time 1787864671.21 total 951.41 last 22.81) - open/close 18661 (time 1787864685.50 total 965.70 last 13.86) - open/close 18787 (time 1787864696.23 total 976.43 last 11.74) - open/close 19019 (time 1787864706.99 total 987.19 last 21.55) - open/close 19389 (time 1787864716.99 total 997.19 last 36.99) - open/close 20000 (time 1787864723.85 total 1004.05 last 89.12) - open/close 20804 (time 1787864733.86 total 1014.06 last 80.31) - open/close 21612 (time 1787864743.87 total 1024.07 last 80.75) - open/close 22383 (time 1787864753.87 total 1034.07 last 77.09) - open/close 23109 (time 1787864763.87 total 1044.08 last 72.56) - open/close 24002 (time 1787864773.88 total 1054.08 last 89.26) - open/close 24731 (time 1787864783.88 total 1064.08 last 72.89) - open/close 25486 (time 1787864793.89 total 1074.09 last 75.42) - open/close 26291 (time 1787864803.90 total 1084.10 last 80.42) - open/close 27076 (time 1787864813.92 total 1094.12 last 78.37) - open/close 27985 (time 1787864823.92 total 1104.12 last 90.85) - open/close 28845 (time 1787864833.92 total 1114.13 last 85.98) - open/close 29772 (time 1787864843.94 total 1124.14 last 92.58) - open/close 30000 (time 1787864847.24 total 1127.44 last 69.12) - open/close 30769 (time 1787864857.25 total 1137.45 last 76.81) - open/close 31519 (time 1787864867.26 total 1147.46 last 74.88) - open/close 32163 (time 1787864877.27 total 1157.47 last 64.37) - open/close 32965 (time 1787864887.27 total 1167.47 last 80.17) - open/close 33824 (time 1787864897.27 total 1177.47 last 85.90) - open/close 34606 (time 1787864907.27 total 1187.47 last 78.20) - open/close 35345 (time 1787864917.28 total 1197.48 last 73.83) - open/close 36187 (time 1787864927.28 total 1207.49 last 84.19) - open/close 36850 (time 1787864937.29 total 1217.49 last 66.26) - open/close 37689 (time 1787864947.30 total 1227.50 last 83.80) - open/close 38466 (time 1787864957.34 total 1237.54 last 77.39) - open/close 39160 (time 1787864967.36 total 1247.56 last 69.27) - open/close 39936 (time 1787864977.39 total 1257.59 last 77.39) - open/close 40718 (time 1787864987.39 total 1267.59 last 78.17) - open/close 41532 (time 1787864997.40 total 1277.60 last 81.32) - open/close 42384 (time 1787865007.42 total 1287.62 last 85.08) - open/close 43257 (time 1787865017.42 total 1297.62 last 87.28) - open/close 44057 (time 1787865027.42 total 1307.62 last 80.00) - open/close 44884 (time 1787865037.42 total 1317.62 last 82.68) - open/close 45690 (time 1787865047.43 total 1327.63 last 80.50) - open/close 46466 (time 1787865057.44 total 1337.64 last 77.57) - open/close 47374 (time 1787865067.44 total 1347.64 last 90.78) - open/close 48296 (time 1787865077.45 total 1357.65 last 92.13) - open/close 49121 (time 1787865087.45 total 1367.65 last 82.46) - open/close 49987 (time 1787865097.47 total 1377.67 last 86.43) total: 50000 open/close in 1378.00 seconds: 36.28 ops/second - unlinked 0 (time 1787865103 ; total 0 ; last 0) - unlinked 10000 (time 1787865241 ; total 138 ; last 138) - unlinked 20000 (time 1787865374 ; total 271 ; last 133) - unlinked 30000 (time 1787865505 ; total 402 ; last 131) - unlinked 40000 (time 1787865636 ; total 533 ; last 131) total: 50000 unlinks in 660 seconds: 75.757576 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg139-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg139-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg139-server stop mds service on oleg139-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg139-server unloading modules via unload_modules_local on: 'oleg139-server' oleg139-server: oleg139-server.virtnet: executing unload_modules_local oleg139-server: modules unloaded. pdsh@oleg139-client: oleg139-client: ssh exited with exit code 2 pdsh@oleg139-client: oleg139-server: ssh exited with exit code 2 pdsh@oleg139-client: oleg139-server: ssh exited with exit code 2 pdsh@oleg139-client: oleg139-client: ssh exited with exit code 2