== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:30:35 (1787193035) start mds service on oleg601-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg601-server' oleg601-server: oleg601-server.virtnet: executing load_modules_local oleg601-server: Loading modules from /home/green/git/lustre-release/lustre oleg601-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg601-server: detected 4 online CPUs by sysfs oleg601-server: MODOPTS_LIBCFS= oleg601-server: Force libcfs to create 2 CPU partitions oleg601-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg601-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg601-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg601-server: oleg601-server.virtnet: executing set_default_debug -1 all pdsh@oleg601-client: oleg601-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg601-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg601-server: oleg601-server.virtnet: executing set_default_debug -1 all pdsh@oleg601-client: oleg601-server: ssh exited with exit code 1 Started lustre-OST0000 oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg601-client.virtnet: -o user_xattr,flock 192.168.206.101@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 234 (time 1787193112.97 total 10.96 last 21.36) - open/close 474 (time 1787193122.99 total 20.97 last 23.96) - open/close 663 (time 1787193133.06 total 31.04 last 18.77) - open/close 919 (time 1787193143.07 total 41.05 last 25.57) - open/close 1004 (time 1787193153.08 total 51.06 last 8.49) - open/close 1252 (time 1787193163.09 total 61.08 last 24.75) - open/close 1441 (time 1787193173.92 total 71.90 last 17.47) - open/close 1698 (time 1787193184.96 total 82.95 last 23.26) - open/close 1912 (time 1787193195.32 total 93.31 last 20.66) - open/close 2142 (time 1787193205.36 total 103.34 last 22.93) - open/close 2343 (time 1787193215.37 total 113.35 last 20.08) - open/close 2585 (time 1787193225.54 total 123.53 last 23.78) - open/close 2816 (time 1787193235.54 total 133.53 last 23.09) - open/close 3014 (time 1787193245.76 total 143.75 last 19.38) - open/close 3195 (time 1787193255.89 total 153.87 last 17.88) - open/close 3437 (time 1787193265.89 total 163.88 last 24.18) - open/close 3675 (time 1787193276.18 total 174.16 last 23.14) - open/close 3900 (time 1787193286.23 total 184.21 last 22.40) - open/close 4123 (time 1787193296.40 total 194.39 last 21.91) - open/close 4351 (time 1787193306.41 total 204.40 last 22.79) - open/close 4563 (time 1787193316.94 total 214.93 last 20.13) - open/close 4801 (time 1787193326.94 total 224.93 last 23.79) - open/close 5050 (time 1787193337.67 total 235.66 last 23.21) - open/close 5312 (time 1787193347.83 total 245.82 last 25.78) - open/close 5516 (time 1787193358.17 total 256.15 last 19.74) - open/close 5741 (time 1787193368.51 total 266.50 last 21.75) - open/close 5995 (time 1787193378.52 total 276.51 last 25.38) - open/close 6204 (time 1787193388.53 total 286.51 last 20.89) - open/close 6473 (time 1787193398.53 total 296.52 last 26.88) - open/close 6763 (time 1787193408.54 total 306.52 last 28.99) - open/close 6964 (time 1787193419.21 total 317.20 last 18.84) - open/close 7173 (time 1787193429.22 total 327.21 last 20.87) - open/close 7469 (time 1787193439.23 total 337.22 last 29.58) - open/close 7662 (time 1787193449.41 total 347.39 last 18.97) - open/close 7896 (time 1787193459.79 total 357.77 last 22.54) - open/close 8136 (time 1787193469.82 total 367.80 last 23.94) - open/close 8431 (time 1787193479.82 total 377.81 last 29.47) - open/close 8660 (time 1787193489.83 total 387.82 last 22.88) - open/close 8898 (time 1787193500.62 total 398.61 last 22.06) - open/close 9102 (time 1787193511.25 total 409.23 last 19.20) - open/close 9338 (time 1787193521.25 total 419.23 last 23.60) - open/close 9552 (time 1787193531.25 total 429.24 last 21.39) - open/close 9772 (time 1787193541.38 total 439.37 last 21.72) - open/close 10000 (time 1787193547.76 total 445.75 last 35.72) - open/close 10208 (time 1787193558.21 total 456.20 last 19.91) - open/close 10487 (time 1787193568.22 total 466.21 last 27.87) - open/close 10698 (time 1787193578.38 total 476.37 last 20.77) - open/close 10810 (time 1787193589.22 total 487.21 last 10.33) - open/close 10917 (time 1787193599.25 total 497.23 last 10.67) - open/close 11065 (time 1787193609.27 total 507.25 last 14.77) - open/close 11210 (time 1787193619.27 total 517.26 last 14.49) - open/close 11446 (time 1787193630.03 total 528.02 last 21.93) - open/close 11744 (time 1787193640.33 total 538.31 last 28.96) - open/close 12012 (time 1787193651.10 total 549.08 last 24.87) - open/close 12234 (time 1787193661.57 total 559.55 last 21.21) - open/close 12445 (time 1787193671.60 total 569.58 last 21.04) - open/close 12686 (time 1787193681.99 total 579.98 last 23.18) - open/close 12919 (time 1787193692.12 total 590.10 last 23.01) - open/close 13181 (time 1787193702.12 total 600.10 last 26.20) - open/close 13424 (time 1787193712.29 total 610.27 last 23.89) - open/close 13688 (time 1787193722.30 total 620.28 last 26.37) - open/close 13826 (time 1787193732.51 total 630.50 last 13.51) - open/close 14146 (time 1787193743.47 total 641.45 last 29.22) - open/close 14409 (time 1787193753.47 total 651.45 last 26.30) - open/close 14591 (time 1787193763.86 total 661.84 last 17.52) - open/close 14866 (time 1787193773.87 total 671.85 last 27.47) - open/close 15061 (time 1787193783.88 total 681.87 last 19.48) - open/close 15269 (time 1787193794.19 total 692.17 last 20.18) - open/close 15495 (time 1787193804.43 total 702.41 last 22.07) - open/close 15708 (time 1787193814.55 total 712.53 last 21.05) - open/close 15902 (time 1787193824.91 total 722.90 last 18.72) - open/close 16096 (time 1787193835.59 total 733.58 last 18.16) - open/close 16282 (time 1787193846.08 total 744.07 last 17.73) - open/close 16528 (time 1787193856.10 total 754.08 last 24.56) - open/close 16752 (time 1787193866.18 total 764.17 last 22.21) - open/close 17044 (time 1787193876.44 total 774.42 last 28.48) - open/close 17305 (time 1787193887.23 total 785.22 last 24.18) - open/close 17488 (time 1787193897.68 total 795.66 last 17.52) - open/close 17715 (time 1787193907.68 total 805.67 last 22.68) - open/close 17928 (time 1787193918.65 total 816.64 last 19.42) - open/close 18147 (time 1787193928.66 total 826.65 last 21.88) - open/close 18387 (time 1787193939.53 total 837.51 last 22.08) - open/close 18576 (time 1787193949.54 total 847.53 last 18.88) - open/close 18823 (time 1787193960.46 total 858.44 last 22.62) - open/close 19096 (time 1787193971.35 total 869.33 last 25.08) - open/close 19738 (time 1787193981.36 total 879.35 last 64.10) - open/close 20000 (time 1787193985.05 total 883.04 last 70.99) - open/close 20697 (time 1787193995.07 total 893.05 last 69.59) - open/close 21425 (time 1787194005.08 total 903.06 last 72.74) - open/close 22133 (time 1787194015.08 total 913.06 last 70.79) - open/close 22822 (time 1787194025.09 total 923.07 last 68.83) - open/close 23545 (time 1787194035.09 total 933.08 last 72.26) - open/close 24260 (time 1787194045.10 total 943.08 last 71.49) - open/close 24971 (time 1787194055.11 total 953.10 last 70.99) - open/close 25696 (time 1787194065.12 total 963.10 last 72.47) - open/close 26314 (time 1787194075.12 total 973.11 last 61.76) - open/close 27014 (time 1787194085.14 total 983.12 last 69.91) - open/close 27636 (time 1787194095.15 total 993.13 last 62.14) - open/close 28319 (time 1787194105.17 total 1003.16 last 68.12) - open/close 29030 (time 1787194115.18 total 1013.16 last 71.04) - open/close 29770 (time 1787194125.18 total 1023.17 last 74.00) - open/close 30000 (time 1787194127.94 total 1025.92 last 83.34) - open/close 30695 (time 1787194137.95 total 1035.93 last 69.46) - open/close 31386 (time 1787194147.95 total 1045.94 last 69.05) - open/close 32000 (time 1787194157.97 total 1055.95 last 61.33) - open/close 32499 (time 1787194167.99 total 1065.98 last 49.77) - open/close 33115 (time 1787194177.99 total 1075.98 last 61.58) - open/close 33775 (time 1787194188.00 total 1085.98 last 65.97) - open/close 34535 (time 1787194198.00 total 1095.99 last 75.98) - open/close 35247 (time 1787194208.00 total 1105.99 last 71.19) - open/close 35980 (time 1787194218.01 total 1116.00 last 73.25) - open/close 36696 (time 1787194228.02 total 1126.01 last 71.50) - open/close 37422 (time 1787194238.04 total 1136.03 last 72.47) - open/close 38069 (time 1787194248.06 total 1146.04 last 64.60) - open/close 38754 (time 1787194258.07 total 1156.05 last 68.44) - open/close 39481 (time 1787194268.08 total 1166.06 last 72.64) - open/close 40000 (time 1787194274.86 total 1172.85 last 76.47) - open/close 40750 (time 1787194284.88 total 1182.86 last 74.90) - open/close 41462 (time 1787194294.88 total 1192.87 last 71.15) - open/close 42257 (time 1787194304.90 total 1202.88 last 79.40) - open/close 43011 (time 1787194314.91 total 1212.89 last 75.33) - open/close 43763 (time 1787194324.91 total 1222.90 last 75.13) - open/close 44527 (time 1787194334.92 total 1232.90 last 76.36) - open/close 45214 (time 1787194344.92 total 1242.91 last 68.68) - open/close 46035 (time 1787194354.93 total 1252.92 last 82.03) - open/close 46820 (time 1787194364.93 total 1262.92 last 78.49) - open/close 47697 (time 1787194374.94 total 1272.92 last 87.67) - open/close 48467 (time 1787194384.94 total 1282.93 last 76.93) - open/close 49248 (time 1787194394.96 total 1292.94 last 78.02) total: 50000 open/close in 1302.86 seconds: 38.38 ops/second - unlinked 0 (time 1787194409 ; total 0 ; last 0) - unlinked 10000 (time 1787194544 ; total 135 ; last 135) - unlinked 20000 (time 1787194686 ; total 277 ; last 142) - unlinked 30000 (time 1787194820 ; total 411 ; last 134) - unlinked 40000 (time 1787194956 ; total 547 ; last 136) total: 50000 unlinks in 683 seconds: 73.206444 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg601-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg601-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg601-server stop mds service on oleg601-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg601-server unloading modules via unload_modules_local on: 'oleg601-server' oleg601-server: oleg601-server.virtnet: executing unload_modules_local oleg601-server: modules unloaded. pdsh@oleg601-client: oleg601-client: ssh exited with exit code 2 pdsh@oleg601-client: oleg601-server: ssh exited with exit code 2 pdsh@oleg601-client: oleg601-server: ssh exited with exit code 2 pdsh@oleg601-client: oleg601-client: ssh exited with exit code 2