== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 04:56:20 (1779872180) start mds service on oleg108-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg108-server' oleg108-server: oleg108-server.virtnet: executing load_modules_local oleg108-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg108-server: Loading modules from /home/green/git/lustre-release/lustre oleg108-server: detected 4 online CPUs by sysfs oleg108-server: MODOPTS_LIBCFS= oleg108-server: Force libcfs to create 2 CPU partitions oleg108-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg108-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg108-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg108-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-OST0000 oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg108-client.virtnet: -o user_xattr,flock 192.168.201.108@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 70 (time 1779872269.28 total 10.01 last 7.00) - open/close 187 (time 1779872280.15 total 20.88 last 10.76) - open/close 335 (time 1779872292.97 total 33.70 last 11.54) - open/close 520 (time 1779872303.21 total 43.94 last 18.07) - open/close 767 (time 1779872313.21 total 53.94 last 24.70) - open/close 1009 (time 1779872323.23 total 63.96 last 24.16) - open/close 1215 (time 1779872333.25 total 73.97 last 20.57) - open/close 1489 (time 1779872343.26 total 83.99 last 27.36) - open/close 1624 (time 1779872353.71 total 94.43 last 12.92) - open/close 1691 (time 1779872363.82 total 104.55 last 6.62) - open/close 1916 (time 1779872374.75 total 115.48 last 20.59) - open/close 2131 (time 1779872385.34 total 126.06 last 20.31) - open/close 2346 (time 1779872396.20 total 136.93 last 19.79) - open/close 2535 (time 1779872406.70 total 147.43 last 18.00) - open/close 2805 (time 1779872417.38 total 158.11 last 25.28) - open/close 3032 (time 1779872427.41 total 168.14 last 22.64) - open/close 3109 (time 1779872438.69 total 179.42 last 6.82) - open/close 3289 (time 1779872449.63 total 190.36 last 16.46) - open/close 3529 (time 1779872459.64 total 200.37 last 23.97) - open/close 3757 (time 1779872469.65 total 210.38 last 22.78) - open/close 3908 (time 1779872480.36 total 221.09 last 14.10) - open/close 4039 (time 1779872491.29 total 232.02 last 11.99) - open/close 4234 (time 1779872502.34 total 243.06 last 17.66) - open/close 4428 (time 1779872512.67 total 253.40 last 18.77) - open/close 4495 (time 1779872522.69 total 263.42 last 6.68) - open/close 4634 (time 1779872533.33 total 274.06 last 13.06) - open/close 4789 (time 1779872544.23 total 284.96 last 14.23) - open/close 4921 (time 1779872555.25 total 295.98 last 11.98) - open/close 5055 (time 1779872565.91 total 306.64 last 12.57) - open/close 5303 (time 1779872575.94 total 316.67 last 24.72) - open/close 5501 (time 1779872586.67 total 327.39 last 18.46) - open/close 5634 (time 1779872596.77 total 337.49 last 13.17) - open/close 5747 (time 1779872606.78 total 347.51 last 11.29) - open/close 5958 (time 1779872616.79 total 357.51 last 21.09) - open/close 6194 (time 1779872627.38 total 368.10 last 22.28) - open/close 6433 (time 1779872637.39 total 378.11 last 23.88) - open/close 6554 (time 1779872648.60 total 389.32 last 10.79) - open/close 6703 (time 1779872660.11 total 400.84 last 12.94) - open/close 6889 (time 1779872672.66 total 413.39 last 14.82) - open/close 7006 (time 1779872682.67 total 423.40 last 11.69) - open/close 7091 (time 1779872693.32 total 434.04 last 7.98) - open/close 7224 (time 1779872703.35 total 444.08 last 13.26) - open/close 7390 (time 1779872714.34 total 455.07 last 15.11) - open/close 7518 (time 1779872733.37 total 474.10 last 6.73) - open/close 7731 (time 1779872744.15 total 484.88 last 19.75) - open/close 7957 (time 1779872754.86 total 495.59 last 21.11) - open/close 8208 (time 1779872767.67 total 508.40 last 19.60) - open/close 8485 (time 1779872780.26 total 520.99 last 22.00) - open/close 8596 (time 1779872791.07 total 531.80 last 10.27) - open/close 8724 (time 1779872801.07 total 541.80 last 12.80) - open/close 8959 (time 1779872811.09 total 551.81 last 23.47) - open/close 9202 (time 1779872821.66 total 562.38 last 22.99) - open/close 9339 (time 1779872831.91 total 572.63 last 13.37) - open/close 9538 (time 1779872842.59 total 583.31 last 18.63) - open/close 9697 (time 1779872853.62 total 594.35 last 14.41) - open/close 9805 (time 1779872863.62 total 604.35 last 10.79) - open/close 9934 (time 1779872875.03 total 615.76 last 11.31) - open/close 10000 (time 1779872879.37 total 620.10 last 15.21) - open/close 10253 (time 1779872889.96 total 630.69 last 23.89) - open/close 10386 (time 1779872900.11 total 640.84 last 13.11) - open/close 10627 (time 1779872911.12 total 651.85 last 21.89) - open/close 10792 (time 1779872923.17 total 663.90 last 13.69) - open/close 10957 (time 1779872933.18 total 673.90 last 16.50) - open/close 11120 (time 1779872943.96 total 684.69 last 15.11) - open/close 11251 (time 1779872954.16 total 694.89 last 12.85) - open/close 11455 (time 1779872964.78 total 705.51 last 19.20) - open/close 11701 (time 1779872975.38 total 716.11 last 23.21) - open/close 11914 (time 1779872986.08 total 726.81 last 19.91) - open/close 12027 (time 1779872996.90 total 737.62 last 10.45) - open/close 12258 (time 1779873007.22 total 747.95 last 22.38) - open/close 12494 (time 1779873017.24 total 757.97 last 23.54) - open/close 12727 (time 1779873027.81 total 768.54 last 22.04) - open/close 12965 (time 1779873038.70 total 779.43 last 21.87) - open/close 13149 (time 1779873048.70 total 789.43 last 18.40) - open/close 13324 (time 1779873058.70 total 799.43 last 17.49) - open/close 13456 (time 1779873068.82 total 809.54 last 13.06) - open/close 13638 (time 1779873079.34 total 820.07 last 17.29) - open/close 13839 (time 1779873089.34 total 830.07 last 20.09) - open/close 14031 (time 1779873099.82 total 840.55 last 18.32) - open/close 14221 (time 1779873109.85 total 850.58 last 18.95) - open/close 14275 (time 1779873120.31 total 861.03 last 5.16) - open/close 14487 (time 1779873130.97 total 871.70 last 19.87) - open/close 14655 (time 1779873141.72 total 882.45 last 15.63) - open/close 14793 (time 1779873151.82 total 892.55 last 13.67) - open/close 15035 (time 1779873161.84 total 902.57 last 24.15) - open/close 15164 (time 1779873172.20 total 912.93 last 12.45) - open/close 15385 (time 1779873182.21 total 922.94 last 22.08) - open/close 15445 (time 1779873192.85 total 933.58 last 5.64) - open/close 15525 (time 1779873203.48 total 944.20 last 7.53) - open/close 15688 (time 1779873213.49 total 954.21 last 16.28) - open/close 15786 (time 1779873224.45 total 965.18 last 8.94) - open/close 16034 (time 1779873234.98 total 975.70 last 23.56) - open/close 16191 (time 1779873246.01 total 986.73 last 14.23) - open/close 16360 (time 1779873256.24 total 996.97 last 16.51) - open/close 16564 (time 1779873266.24 total 1006.97 last 20.40) - open/close 16667 (time 1779873277.28 total 1018.00 last 9.33) - open/close 16789 (time 1779873287.28 total 1028.01 last 12.19) - open/close 16916 (time 1779873297.49 total 1038.22 last 12.44) - open/close 17059 (time 1779873307.52 total 1048.25 last 14.26) - open/close 17588 (time 1779873317.53 total 1058.26 last 52.86) - open/close 18221 (time 1779873327.54 total 1068.27 last 63.24) - open/close 18853 (time 1779873337.55 total 1078.28 last 63.13) - open/close 19578 (time 1779873347.55 total 1088.28 last 72.47) - open/close 20000 (time 1779873353.74 total 1094.47 last 68.17) - open/close 20725 (time 1779873363.76 total 1104.49 last 72.39) - open/close 21384 (time 1779873373.77 total 1114.50 last 65.84) - open/close 21988 (time 1779873383.79 total 1124.51 last 60.31) - open/close 22598 (time 1779873393.80 total 1134.52 last 60.94) - open/close 23329 (time 1779873403.81 total 1144.53 last 73.03) - open/close 24053 (time 1779873413.82 total 1154.54 last 72.32) - open/close 24678 (time 1779873423.83 total 1164.55 last 62.45) - open/close 25411 (time 1779873433.84 total 1174.56 last 73.22) - open/close 26115 (time 1779873443.84 total 1184.57 last 70.38) - open/close 26794 (time 1779873453.84 total 1194.57 last 67.87) - open/close 27502 (time 1779873463.85 total 1204.57 last 70.79) - open/close 28217 (time 1779873473.86 total 1214.58 last 71.41) - open/close 28808 (time 1779873483.86 total 1224.59 last 59.06) - open/close 29356 (time 1779873493.88 total 1234.61 last 54.71) - open/close 29913 (time 1779873503.88 total 1244.61 last 55.69) - open/close 30549 (time 1779873513.88 total 1254.61 last 63.60) - open/close 31244 (time 1779873523.88 total 1264.61 last 69.49) - open/close 31828 (time 1779873533.89 total 1274.61 last 58.39) - open/close 32439 (time 1779873543.89 total 1284.62 last 61.04) - open/close 33108 (time 1779873553.92 total 1294.64 last 66.76) - open/close 33675 (time 1779873563.92 total 1304.65 last 56.67) - open/close 34289 (time 1779873573.93 total 1314.66 last 61.33) - open/close 34940 (time 1779873583.94 total 1324.67 last 65.03) - open/close 35599 (time 1779873593.94 total 1334.67 last 65.89) - open/close 36078 (time 1779873603.98 total 1344.70 last 47.74) - open/close 36489 (time 1779873614.00 total 1354.73 last 41.01) - open/close 36968 (time 1779873624.00 total 1364.73 last 47.87) - open/close 37626 (time 1779873634.01 total 1374.74 last 65.74) - open/close 38227 (time 1779873644.02 total 1384.74 last 60.07) - open/close 38979 (time 1779873654.03 total 1394.75 last 75.13) - open/close 39601 (time 1779873664.03 total 1404.76 last 62.19) - open/close 40000 (time 1779873671.12 total 1411.85 last 56.24) - open/close 40719 (time 1779873681.14 total 1421.87 last 71.77) - open/close 41390 (time 1779873691.14 total 1431.87 last 67.10) - open/close 42092 (time 1779873701.16 total 1441.88 last 70.11) - open/close 42779 (time 1779873711.16 total 1451.89 last 68.64) - open/close 43482 (time 1779873721.18 total 1461.91 last 70.19) - open/close 44158 (time 1779873731.18 total 1471.91 last 67.58) - open/close 44865 (time 1779873741.19 total 1481.91 last 70.69) - open/close 45687 (time 1779873751.19 total 1491.92 last 82.16) - open/close 46370 (time 1779873761.20 total 1501.93 last 68.20) - open/close 47068 (time 1779873771.21 total 1511.94 last 69.76) - open/close 47792 (time 1779873781.23 total 1521.96 last 72.21) - open/close 48558 (time 1779873791.24 total 1531.97 last 76.53) - open/close 49260 (time 1779873801.25 total 1541.98 last 70.17) - open/close 49998 (time 1779873811.26 total 1551.98 last 73.74) total: 50000 open/close in 1552.00 seconds: 32.22 ops/second - unlinked 0 (time 1779873815 ; total 0 ; last 0) - unlinked 10000 (time 1779873951 ; total 136 ; last 136) - unlinked 20000 (time 1779874110 ; total 295 ; last 159) - unlinked 30000 (time 1779874275 ; total 460 ; last 165) - unlinked 40000 (time 1779874448 ; total 633 ; last 173) total: 50000 unlinks in 810 seconds: 61.728394 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg108-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg108-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg108-server stop mds service on oleg108-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg108-server unloading modules via unload_modules_local on: 'oleg108-server' oleg108-server: oleg108-server.virtnet: executing unload_modules_local oleg108-server: modules unloaded. pdsh@oleg108-client: oleg108-server: ssh exited with exit code 2 pdsh@oleg108-client: oleg108-client: ssh exited with exit code 2 pdsh@oleg108-client: oleg108-client: ssh exited with exit code 2 pdsh@oleg108-client: oleg108-server: ssh exited with exit code 2