== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 07:44:59 (1789127099) start mds service on oleg209-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg209-server' oleg209-server: oleg209-server.virtnet: executing load_modules_local oleg209-server: Loading modules from /home/green/git/lustre-release/lustre oleg209-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg209-server: detected 4 online CPUs by sysfs oleg209-server: MODOPTS_LIBCFS= oleg209-server: Force libcfs to create 2 CPU partitions oleg209-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg209-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg209-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg209-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0000 oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 221 (time 1789127198.84 total 10.46 last 21.13) - open/close 463 (time 1789127208.86 total 20.47 last 24.17) - open/close 640 (time 1789127219.68 total 31.29 last 16.36) - open/close 757 (time 1789127229.91 total 41.53 last 11.43) - open/close 845 (time 1789127240.22 total 51.83 last 8.54) - open/close 1065 (time 1789127250.23 total 61.84 last 21.98) - open/close 1254 (time 1789127260.24 total 71.86 last 18.87) - open/close 1524 (time 1789127270.28 total 81.89 last 26.90) - open/close 1705 (time 1789127280.29 total 91.91 last 18.07) - open/close 1961 (time 1789127290.59 total 102.21 last 24.85) - open/close 2192 (time 1789127300.60 total 112.22 last 23.08) - open/close 2372 (time 1789127310.61 total 122.23 last 17.99) - open/close 2569 (time 1789127320.73 total 132.34 last 19.47) - open/close 2831 (time 1789127330.73 total 142.35 last 26.19) - open/close 3006 (time 1789127341.27 total 152.88 last 16.61) - open/close 3247 (time 1789127351.59 total 163.20 last 23.36) - open/close 3500 (time 1789127361.59 total 173.21 last 25.29) - open/close 3663 (time 1789127372.41 total 184.03 last 15.06) - open/close 3911 (time 1789127383.19 total 194.81 last 23.01) - open/close 4152 (time 1789127393.37 total 204.99 last 23.68) - open/close 4425 (time 1789127403.38 total 215.00 last 27.26) - open/close 4658 (time 1789127413.40 total 225.01 last 23.27) - open/close 4837 (time 1789127423.44 total 235.05 last 17.83) - open/close 4995 (time 1789127433.62 total 245.24 last 15.52) - open/close 5254 (time 1789127443.62 total 255.24 last 25.89) - open/close 5529 (time 1789127454.07 total 265.68 last 26.33) - open/close 5670 (time 1789127464.07 total 275.68 last 14.10) - open/close 5820 (time 1789127474.43 total 286.05 last 14.48) - open/close 5989 (time 1789127484.44 total 296.06 last 16.88) - open/close 6210 (time 1789127495.06 total 306.68 last 20.81) - open/close 6409 (time 1789127505.07 total 316.69 last 19.88) - open/close 6647 (time 1789127517.93 total 329.54 last 18.51) - open/close 6840 (time 1789127527.93 total 339.55 last 19.29) - open/close 6912 (time 1789127538.05 total 349.67 last 7.11) - open/close 7137 (time 1789127548.12 total 359.74 last 22.33) - open/close 7462 (time 1789127558.13 total 369.74 last 32.50) - open/close 7703 (time 1789127568.13 total 379.75 last 24.09) - open/close 7859 (time 1789127579.20 total 390.81 last 14.10) - open/close 8105 (time 1789127589.23 total 400.84 last 24.53) - open/close 8307 (time 1789127599.52 total 411.13 last 19.63) - open/close 8527 (time 1789127609.62 total 421.24 last 21.77) - open/close 8646 (time 1789127619.63 total 431.25 last 11.89) - open/close 8811 (time 1789127629.63 total 441.25 last 16.50) - open/close 9023 (time 1789127640.12 total 451.73 last 20.23) - open/close 9184 (time 1789127650.20 total 461.82 last 15.97) - open/close 9331 (time 1789127660.87 total 472.48 last 13.78) - open/close 9487 (time 1789127670.87 total 482.49 last 15.59) - open/close 9624 (time 1789127682.39 total 494.01 last 11.89) - open/close 9758 (time 1789127692.57 total 504.19 last 13.16) - open/close 9853 (time 1789127702.64 total 514.26 last 9.43) - open/close 10000 (time 1789127710.58 total 522.19 last 18.53) - open/close 10119 (time 1789127720.59 total 532.20 last 11.89) - open/close 10337 (time 1789127730.60 total 542.22 last 21.77) - open/close 10488 (time 1789127741.36 total 552.98 last 14.03) - open/close 10664 (time 1789127751.38 total 563.00 last 17.56) - open/close 10834 (time 1789127761.40 total 573.02 last 16.97) - open/close 11033 (time 1789127773.72 total 585.33 last 16.16) - open/close 11163 (time 1789127783.84 total 595.46 last 12.84) - open/close 11442 (time 1789127794.48 total 606.10 last 26.23) - open/close 11687 (time 1789127805.28 total 616.89 last 22.69) - open/close 11798 (time 1789127815.29 total 626.91 last 11.09) - open/close 11932 (time 1789127829.60 total 641.22 last 9.36) - open/close 12160 (time 1789127839.62 total 651.23 last 22.77) - open/close 12405 (time 1789127849.78 total 661.39 last 24.11) - open/close 12610 (time 1789127860.09 total 671.70 last 19.89) - open/close 12811 (time 1789127870.43 total 682.05 last 19.43) - open/close 13013 (time 1789127880.44 total 692.05 last 20.19) - open/close 13251 (time 1789127891.12 total 702.74 last 22.28) - open/close 13437 (time 1789127902.13 total 713.75 last 16.90) - open/close 13694 (time 1789127912.14 total 723.76 last 25.66) - open/close 13927 (time 1789127922.62 total 734.24 last 22.24) - open/close 14154 (time 1789127933.49 total 745.11 last 20.88) - open/close 14414 (time 1789127943.50 total 755.12 last 25.98) - open/close 14619 (time 1789127953.64 total 765.25 last 20.23) - open/close 14838 (time 1789127965.04 total 776.66 last 19.20) - open/close 14956 (time 1789127975.05 total 786.67 last 11.79) - open/close 15161 (time 1789127985.44 total 797.06 last 19.73) - open/close 15363 (time 1789127995.93 total 807.55 last 19.25) - open/close 15589 (time 1789128005.94 total 817.56 last 22.59) - open/close 15828 (time 1789128015.98 total 827.60 last 23.81) - open/close 16019 (time 1789128026.00 total 837.62 last 19.05) - open/close 16240 (time 1789128037.92 total 849.54 last 18.54) - open/close 16427 (time 1789128050.18 total 861.79 last 15.26) - open/close 16583 (time 1789128060.23 total 871.84 last 15.52) - open/close 16715 (time 1789128071.02 total 882.63 last 12.23) - open/close 16793 (time 1789128081.97 total 893.59 last 7.12) - open/close 17008 (time 1789128091.98 total 903.60 last 21.47) - open/close 17244 (time 1789128102.69 total 914.30 last 22.05) - open/close 17419 (time 1789128115.35 total 926.97 last 13.82) - open/close 17646 (time 1789128125.40 total 937.01 last 22.61) - open/close 17880 (time 1789128135.49 total 947.10 last 23.19) - open/close 18130 (time 1789128145.65 total 957.27 last 24.60) - open/close 18372 (time 1789128156.34 total 967.96 last 22.63) - open/close 18665 (time 1789128166.34 total 977.96 last 29.29) - open/close 19359 (time 1789128176.35 total 987.96 last 69.37) - open/close 20000 (time 1789128185.50 total 997.12 last 70.02) - open/close 20682 (time 1789128195.51 total 1007.12 last 68.17) - open/close 21393 (time 1789128205.51 total 1017.13 last 71.05) - open/close 22100 (time 1789128215.52 total 1027.13 last 70.67) - open/close 22716 (time 1789128225.53 total 1037.15 last 61.52) - open/close 23428 (time 1789128235.54 total 1047.16 last 71.12) - open/close 24174 (time 1789128245.56 total 1057.17 last 74.50) - open/close 24901 (time 1789128255.56 total 1067.18 last 72.65) - open/close 25741 (time 1789128265.57 total 1077.19 last 83.95) - open/close 26538 (time 1789128275.61 total 1087.23 last 79.36) - open/close 27287 (time 1789128285.62 total 1097.23 last 74.85) - open/close 27969 (time 1789128295.62 total 1107.24 last 68.19) - open/close 28757 (time 1789128305.63 total 1117.24 last 78.74) - open/close 29650 (time 1789128315.63 total 1127.25 last 89.27) - open/close 30000 (time 1789128319.95 total 1131.56 last 81.12) - open/close 30658 (time 1789128329.95 total 1141.56 last 65.80) - open/close 31361 (time 1789128339.95 total 1151.57 last 70.27) - open/close 32032 (time 1789128349.96 total 1161.57 last 67.05) - open/close 32764 (time 1789128359.96 total 1171.58 last 73.14) - open/close 33527 (time 1789128369.97 total 1181.59 last 76.27) - open/close 34115 (time 1789128379.97 total 1191.59 last 58.79) - open/close 34793 (time 1789128389.98 total 1201.59 last 67.76) - open/close 35471 (time 1789128399.98 total 1211.60 last 67.78) - open/close 36196 (time 1789128409.99 total 1221.60 last 72.45) - open/close 36973 (time 1789128420.00 total 1231.61 last 77.61) - open/close 37725 (time 1789128430.00 total 1241.62 last 75.17) - open/close 38369 (time 1789128440.00 total 1251.62 last 64.39) - open/close 39155 (time 1789128450.01 total 1261.63 last 78.52) - open/close 39888 (time 1789128460.01 total 1271.63 last 73.28) - open/close 40524 (time 1789128470.02 total 1281.63 last 63.58) - open/close 41202 (time 1789128480.02 total 1291.63 last 67.79) - open/close 41886 (time 1789128490.03 total 1301.65 last 68.30) - open/close 42527 (time 1789128500.05 total 1311.66 last 64.01) - open/close 43089 (time 1789128510.06 total 1321.68 last 56.13) - open/close 43779 (time 1789128520.08 total 1331.69 last 68.88) - open/close 44521 (time 1789128530.08 total 1341.70 last 74.16) - open/close 45227 (time 1789128540.08 total 1351.70 last 70.60) - open/close 46066 (time 1789128550.09 total 1361.71 last 83.82) - open/close 46908 (time 1789128560.10 total 1371.72 last 84.15) - open/close 47618 (time 1789128570.11 total 1381.73 last 70.88) - open/close 48265 (time 1789128580.14 total 1391.75 last 64.56) - open/close 49011 (time 1789128590.15 total 1401.77 last 74.50) - open/close 49735 (time 1789128600.16 total 1411.77 last 72.34) total: 50000 open/close in 1415.16 seconds: 35.33 ops/second - unlinked 0 (time 1789128607 ; total 0 ; last 0) - unlinked 10000 (time 1789128752 ; total 145 ; last 145) - unlinked 20000 (time 1789128911 ; total 304 ; last 159) - unlinked 30000 (time 1789129061 ; total 454 ; last 150) - unlinked 40000 (time 1789129214 ; total 607 ; last 153) total: 50000 unlinks in 770 seconds: 64.935066 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg209-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg209-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg209-server stop mds service on oleg209-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg209-server unloading modules via unload_modules_local on: 'oleg209-server' oleg209-server: oleg209-server.virtnet: executing unload_modules_local oleg209-server: modules unloaded. pdsh@oleg209-client: oleg209-client: ssh exited with exit code 2 pdsh@oleg209-client: oleg209-server: ssh exited with exit code 2 pdsh@oleg209-client: oleg209-client: ssh exited with exit code 2 pdsh@oleg209-client: oleg209-server: ssh exited with exit code 2