== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:58:39 (1788443919) start mds service on oleg107-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg107-server' oleg107-server: oleg107-server.virtnet: executing load_modules_local oleg107-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg107-server: Loading modules from /home/green/git/lustre-release/lustre oleg107-server: detected 4 online CPUs by sysfs oleg107-server: MODOPTS_LIBCFS= oleg107-server: Force libcfs to create 2 CPU partitions oleg107-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg107-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg107-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg107-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-OST0000 oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg107-client.virtnet: -o user_xattr,flock 192.168.201.107@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 78 (time 1788444014.44 total 10.50 last 7.43) - open/close 146 (time 1788444024.49 total 20.56 last 6.76) - open/close 190 (time 1788444036.96 total 33.03 last 3.53) - open/close 335 (time 1788444048.42 total 44.49 last 12.65) - open/close 429 (time 1788444058.43 total 54.49 last 9.40) - open/close 589 (time 1788444069.28 total 65.34 last 14.75) - open/close 711 (time 1788444080.36 total 76.43 last 11.00) - open/close 867 (time 1788444090.97 total 87.04 last 14.71) - open/close 1155 (time 1788444101.61 total 97.67 last 27.08) - open/close 1394 (time 1788444111.62 total 107.68 last 23.88) - open/close 1551 (time 1788444122.13 total 118.20 last 14.93) - open/close 1736 (time 1788444132.14 total 128.21 last 18.48) - open/close 1995 (time 1788444142.15 total 138.21 last 25.89) - open/close 2225 (time 1788444153.12 total 149.18 last 20.97) - open/close 2555 (time 1788444163.86 total 159.93 last 30.72) - open/close 2743 (time 1788444175.66 total 171.72 last 15.94) - open/close 2983 (time 1788444186.60 total 182.66 last 21.93) - open/close 3125 (time 1788444196.85 total 192.91 last 13.86) - open/close 3334 (time 1788444206.85 total 202.92 last 20.88) - open/close 3573 (time 1788444218.28 total 214.35 last 20.91) - open/close 3710 (time 1788444228.64 total 224.70 last 13.23) - open/close 3921 (time 1788444240.86 total 236.92 last 17.27) - open/close 4057 (time 1788444251.05 total 247.11 last 13.34) - open/close 4257 (time 1788444261.98 total 258.04 last 18.30) - open/close 4441 (time 1788444272.96 total 269.02 last 16.76) - open/close 4697 (time 1788444283.93 total 280.00 last 23.32) - open/close 4812 (time 1788444293.94 total 290.01 last 11.49) - open/close 4962 (time 1788444303.94 total 300.01 last 15.00) - open/close 5046 (time 1788444314.53 total 310.60 last 7.93) - open/close 5083 (time 1788444325.16 total 321.22 last 3.48) - open/close 5263 (time 1788444335.41 total 331.47 last 17.56) - open/close 5491 (time 1788444346.52 total 342.59 last 20.51) - open/close 5785 (time 1788444356.54 total 352.60 last 29.34) - open/close 5922 (time 1788444366.55 total 362.62 last 13.68) - open/close 6083 (time 1788444376.56 total 372.63 last 16.08) - open/close 6273 (time 1788444386.85 total 382.91 last 18.47) - open/close 6497 (time 1788444399.18 total 395.24 last 18.17) - open/close 6727 (time 1788444410.85 total 406.91 last 19.71) - open/close 6971 (time 1788444420.87 total 416.94 last 24.35) - open/close 7170 (time 1788444430.90 total 426.96 last 19.85) - open/close 7304 (time 1788444441.83 total 437.89 last 12.26) - open/close 7460 (time 1788444452.44 total 448.51 last 14.69) - open/close 7566 (time 1788444462.58 total 458.64 last 10.46) - open/close 7814 (time 1788444472.58 total 468.65 last 24.79) - open/close 8051 (time 1788444482.60 total 478.66 last 23.66) - open/close 8242 (time 1788444493.47 total 489.53 last 17.57) - open/close 8405 (time 1788444503.71 total 499.77 last 15.91) - open/close 8538 (time 1788444513.71 total 509.77 last 13.30) - open/close 8774 (time 1788444525.97 total 522.03 last 19.25) - open/close 9038 (time 1788444536.58 total 532.64 last 24.89) - open/close 9292 (time 1788444547.43 total 543.50 last 23.39) - open/close 9440 (time 1788444557.95 total 554.01 last 14.08) - open/close 9538 (time 1788444570.46 total 566.52 last 7.84) - open/close 9693 (time 1788444580.96 total 577.03 last 14.75) - open/close 9962 (time 1788444590.97 total 587.03 last 26.88) - open/close 10126 (time 1788444601.87 total 597.94 last 15.04) - open/close 10328 (time 1788444612.39 total 608.45 last 19.22) - open/close 10451 (time 1788444622.91 total 618.97 last 11.69) - open/close 10656 (time 1788444633.18 total 629.25 last 19.95) - open/close 10746 (time 1788444643.94 total 640.01 last 8.37) - open/close 10999 (time 1788444654.89 total 650.96 last 23.10) - open/close 11221 (time 1788444665.42 total 661.48 last 21.09) - open/close 11530 (time 1788444675.42 total 671.49 last 30.89) - open/close 11698 (time 1788444685.47 total 681.54 last 16.71) - open/close 11902 (time 1788444696.23 total 692.30 last 18.96) - open/close 12141 (time 1788444706.25 total 702.31 last 23.87) - open/close 12283 (time 1788444716.44 total 712.50 last 13.94) - open/close 12422 (time 1788444727.47 total 723.53 last 12.60) - open/close 12514 (time 1788444741.16 total 737.23 last 6.72) - open/close 12720 (time 1788444754.23 total 750.29 last 15.76) - open/close 12853 (time 1788444765.12 total 761.18 last 12.22) - open/close 13047 (time 1788444775.12 total 771.19 last 19.39) - open/close 13190 (time 1788444788.00 total 784.07 last 11.10) - open/close 13356 (time 1788444798.82 total 794.89 last 15.34) - open/close 13599 (time 1788444808.84 total 804.90 last 24.27) - open/close 13732 (time 1788444818.85 total 814.91 last 13.29) - open/close 13904 (time 1788444828.85 total 824.92 last 17.19) - open/close 14149 (time 1788444839.13 total 835.20 last 23.83) - open/close 14355 (time 1788444849.46 total 845.53 last 19.94) - open/close 14463 (time 1788444860.53 total 856.60 last 9.76) - open/close 14565 (time 1788444870.94 total 867.01 last 9.80) - open/close 14688 (time 1788444881.64 total 877.71 last 11.49) - open/close 14850 (time 1788444892.21 total 888.28 last 15.33) - open/close 15028 (time 1788444902.23 total 898.29 last 17.77) - open/close 15111 (time 1788444912.76 total 908.83 last 7.88) - open/close 15332 (time 1788444923.82 total 919.88 last 19.99) - open/close 15498 (time 1788444934.63 total 930.69 last 15.35) - open/close 15692 (time 1788444944.66 total 940.72 last 19.34) - open/close 15835 (time 1788444954.66 total 950.72 last 14.30) - open/close 16021 (time 1788444964.67 total 960.73 last 18.59) - open/close 16231 (time 1788444975.73 total 971.79 last 18.98) - open/close 16306 (time 1788444986.72 total 982.78 last 6.82) - open/close 16485 (time 1788444996.74 total 992.80 last 17.87) - open/close 16718 (time 1788445006.74 total 1002.81 last 23.28) - open/close 16906 (time 1788445016.75 total 1012.81 last 18.79) - open/close 17117 (time 1788445027.18 total 1023.24 last 20.23) - open/close 17350 (time 1788445037.18 total 1033.24 last 23.30) - open/close 17521 (time 1788445047.18 total 1043.25 last 17.10) - open/close 18039 (time 1788445057.19 total 1053.25 last 51.77) - open/close 18700 (time 1788445067.19 total 1063.26 last 66.05) - open/close 19390 (time 1788445077.20 total 1073.26 last 68.96) - open/close 20000 (time 1788445084.66 total 1080.72 last 81.79) - open/close 20761 (time 1788445094.67 total 1090.74 last 76.01) - open/close 21665 (time 1788445104.67 total 1100.74 last 90.39) - open/close 22431 (time 1788445114.69 total 1110.75 last 76.46) - open/close 23058 (time 1788445124.70 total 1120.76 last 62.65) - open/close 23865 (time 1788445134.70 total 1130.77 last 80.68) - open/close 24664 (time 1788445144.70 total 1140.77 last 79.87) - open/close 25518 (time 1788445154.71 total 1150.77 last 85.38) - open/close 26205 (time 1788445164.71 total 1160.77 last 68.70) - open/close 27005 (time 1788445174.72 total 1170.78 last 79.93) - open/close 27863 (time 1788445184.73 total 1180.79 last 85.69) - open/close 28720 (time 1788445194.74 total 1190.80 last 85.62) - open/close 29522 (time 1788445204.74 total 1200.81 last 80.18) - open/close 30000 (time 1788445210.67 total 1206.73 last 80.64) - open/close 30669 (time 1788445220.67 total 1216.74 last 66.89) - open/close 31418 (time 1788445230.67 total 1226.74 last 74.89) - open/close 32144 (time 1788445240.69 total 1236.75 last 72.50) - open/close 32681 (time 1788445250.69 total 1246.75 last 53.69) - open/close 33415 (time 1788445260.70 total 1256.76 last 73.33) - open/close 34182 (time 1788445270.71 total 1266.77 last 76.61) - open/close 34945 (time 1788445280.71 total 1276.77 last 76.30) - open/close 35561 (time 1788445290.72 total 1286.78 last 61.55) - open/close 36201 (time 1788445300.72 total 1296.78 last 64.00) - open/close 36786 (time 1788445310.72 total 1306.79 last 58.46) - open/close 37653 (time 1788445320.74 total 1316.80 last 86.57) - open/close 38364 (time 1788445330.75 total 1326.82 last 70.99) - open/close 39147 (time 1788445340.77 total 1336.83 last 78.19) - open/close 39946 (time 1788445350.77 total 1346.84 last 79.87) - open/close 40595 (time 1788445360.77 total 1356.84 last 64.88) - open/close 41222 (time 1788445370.78 total 1366.84 last 62.68) - open/close 41962 (time 1788445380.78 total 1376.85 last 73.95) - open/close 42634 (time 1788445390.80 total 1386.86 last 67.12) - open/close 43231 (time 1788445400.81 total 1396.87 last 59.64) - open/close 43848 (time 1788445410.81 total 1406.87 last 61.69) - open/close 44372 (time 1788445420.82 total 1416.88 last 52.37) - open/close 45112 (time 1788445430.82 total 1426.88 last 73.99) - open/close 45973 (time 1788445440.83 total 1436.89 last 86.02) - open/close 46627 (time 1788445450.83 total 1446.90 last 65.37) - open/close 47403 (time 1788445460.84 total 1456.90 last 77.57) - open/close 48090 (time 1788445470.85 total 1466.91 last 68.61) - open/close 48848 (time 1788445480.85 total 1476.92 last 75.77) - open/close 49566 (time 1788445490.86 total 1486.92 last 71.77) total: 50000 open/close in 1492.13 seconds: 33.51 ops/second - unlinked 0 (time 1788445500 ; total 0 ; last 0) - unlinked 10000 (time 1788445648 ; total 148 ; last 148) - unlinked 20000 (time 1788445782 ; total 282 ; last 134) - unlinked 30000 (time 1788445920 ; total 420 ; last 138) - unlinked 40000 (time 1788446061 ; total 561 ; last 141) total: 50000 unlinks in 693 seconds: 72.150070 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg107-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg107-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg107-server stop mds service on oleg107-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg107-server unloading modules via unload_modules_local on: 'oleg107-server' oleg107-server: oleg107-server.virtnet: executing unload_modules_local oleg107-server: modules unloaded. pdsh@oleg107-client: oleg107-client: ssh exited with exit code 2 pdsh@oleg107-client: oleg107-server: ssh exited with exit code 2 pdsh@oleg107-client: oleg107-client: ssh exited with exit code 2 pdsh@oleg107-client: oleg107-server: ssh exited with exit code 2