== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 15:33:22 (1787340802) start mds service on oleg318-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions oleg318-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg318-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg318-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg318-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-OST0000 oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 215 (time 1787340886.28 total 11.02 last 19.52) - open/close 455 (time 1787340896.29 total 21.03 last 23.97) - open/close 656 (time 1787340906.32 total 31.06 last 20.03) - open/close 863 (time 1787340916.35 total 41.09 last 20.65) - open/close 1119 (time 1787340927.23 total 51.97 last 23.54) - open/close 1351 (time 1787340937.24 total 61.98 last 23.18) - open/close 1529 (time 1787340949.71 total 74.45 last 14.27) - open/close 1725 (time 1787340959.71 total 84.46 last 19.59) - open/close 1931 (time 1787340971.91 total 96.65 last 16.89) - open/close 2145 (time 1787340982.46 total 107.21 last 20.28) - open/close 2359 (time 1787340994.95 total 119.69 last 17.15) - open/close 2601 (time 1787341005.06 total 129.80 last 23.94) - open/close 2761 (time 1787341015.43 total 140.17 last 15.43) - open/close 2907 (time 1787341025.43 total 150.17 last 14.60) - open/close 3138 (time 1787341035.71 total 160.45 last 22.48) - open/close 3395 (time 1787341048.77 total 173.51 last 19.67) - open/close 3564 (time 1787341059.00 total 183.74 last 16.51) - open/close 3832 (time 1787341069.91 total 194.65 last 24.58) - open/close 4052 (time 1787341080.25 total 204.99 last 21.27) - open/close 4303 (time 1787341090.62 total 215.36 last 24.21) - open/close 4538 (time 1787341100.93 total 225.67 last 22.79) - open/close 4719 (time 1787341111.23 total 235.97 last 17.57) - open/close 4869 (time 1787341121.86 total 246.60 last 14.12) - open/close 4964 (time 1787341132.56 total 257.30 last 8.87) - open/close 5187 (time 1787341142.97 total 267.71 last 21.42) - open/close 5304 (time 1787341152.99 total 277.73 last 11.68) - open/close 5483 (time 1787341163.00 total 287.74 last 17.88) - open/close 5711 (time 1787341173.13 total 297.87 last 22.51) - open/close 5937 (time 1787341183.61 total 308.35 last 21.56) - open/close 6192 (time 1787341193.64 total 318.38 last 25.43) - open/close 6430 (time 1787341203.77 total 328.51 last 23.50) - open/close 6577 (time 1787341213.77 total 338.51 last 14.70) - open/close 6724 (time 1787341223.81 total 348.55 last 14.64) - open/close 6899 (time 1787341234.45 total 359.19 last 16.45) - open/close 7012 (time 1787341244.46 total 369.20 last 11.28) - open/close 7149 (time 1787341254.73 total 379.47 last 13.35) - open/close 7330 (time 1787341264.74 total 389.48 last 18.08) - open/close 7546 (time 1787341274.74 total 399.48 last 21.60) - open/close 7757 (time 1787341285.20 total 409.94 last 20.18) - open/close 8001 (time 1787341295.20 total 419.94 last 24.38) - open/close 8214 (time 1787341306.11 total 430.85 last 19.54) - open/close 8411 (time 1787341316.18 total 440.92 last 19.57) - open/close 8560 (time 1787341327.49 total 452.23 last 13.17) - open/close 8760 (time 1787341337.50 total 462.24 last 20.00) - open/close 8936 (time 1787341348.12 total 472.86 last 16.57) - open/close 9097 (time 1787341358.28 total 483.02 last 15.84) - open/close 9275 (time 1787341368.45 total 493.19 last 17.50) - open/close 9440 (time 1787341381.46 total 506.20 last 12.69) - open/close 9678 (time 1787341391.88 total 516.62 last 22.84) - open/close 9792 (time 1787341402.77 total 527.51 last 10.47) - open/close 9926 (time 1787341413.44 total 538.18 last 12.55) - open/close 10000 (time 1787341420.74 total 545.48 last 10.15) - open/close 10170 (time 1787341430.74 total 555.48 last 16.99) - open/close 10337 (time 1787341440.83 total 565.57 last 16.55) - open/close 10423 (time 1787341452.18 total 576.92 last 7.58) - open/close 10669 (time 1787341462.35 total 587.09 last 24.19) - open/close 10901 (time 1787341472.35 total 597.09 last 23.18) - open/close 11112 (time 1787341482.49 total 607.23 last 20.81) - open/close 11346 (time 1787341492.85 total 617.60 last 22.58) - open/close 11552 (time 1787341502.88 total 627.62 last 20.56) - open/close 11708 (time 1787341513.02 total 637.76 last 15.38) - open/close 11784 (time 1787341523.04 total 647.78 last 7.58) - open/close 12029 (time 1787341533.51 total 658.25 last 23.40) - open/close 12238 (time 1787341543.87 total 668.61 last 20.18) - open/close 12482 (time 1787341553.87 total 678.61 last 24.39) - open/close 12692 (time 1787341563.88 total 688.62 last 20.98) - open/close 12914 (time 1787341574.35 total 699.09 last 21.21) - open/close 13162 (time 1787341584.35 total 709.09 last 24.79) - open/close 13369 (time 1787341596.69 total 721.43 last 16.77) - open/close 13577 (time 1787341607.33 total 732.07 last 19.56) - open/close 13809 (time 1787341617.34 total 742.08 last 23.18) - open/close 14055 (time 1787341628.30 total 753.04 last 22.44) - open/close 14234 (time 1787341638.31 total 763.05 last 17.88) - open/close 14477 (time 1787341649.34 total 774.08 last 22.03) - open/close 14671 (time 1787341659.34 total 784.08 last 19.39) - open/close 14837 (time 1787341669.92 total 794.66 last 15.69) - open/close 15073 (time 1787341679.92 total 804.66 last 23.60) - open/close 15275 (time 1787341691.28 total 816.02 last 17.78) - open/close 15504 (time 1787341702.36 total 827.10 last 20.66) - open/close 15728 (time 1787341712.38 total 837.13 last 22.35) - open/close 15840 (time 1787341722.40 total 847.14 last 11.19) - open/close 15979 (time 1787341732.68 total 857.42 last 13.51) - open/close 16066 (time 1787341743.04 total 867.78 last 8.40) - open/close 16228 (time 1787341755.65 total 880.39 last 12.85) - open/close 16487 (time 1787341768.74 total 893.48 last 19.79) - open/close 16602 (time 1787341778.75 total 903.49 last 11.48) - open/close 16789 (time 1787341788.76 total 913.50 last 18.69) - open/close 16951 (time 1787341798.78 total 923.52 last 16.17) - open/close 17175 (time 1787341809.50 total 934.24 last 20.90) - open/close 17392 (time 1787341819.78 total 944.52 last 21.10) - open/close 17717 (time 1787341829.79 total 954.53 last 32.47) - open/close 18375 (time 1787341839.80 total 964.54 last 65.74) - open/close 19120 (time 1787341849.81 total 974.55 last 74.46) - open/close 19777 (time 1787341859.83 total 984.57 last 65.54) - open/close 20000 (time 1787341863.37 total 988.11 last 63.06) - open/close 20667 (time 1787341873.38 total 998.12 last 66.58) - open/close 21362 (time 1787341883.39 total 1008.13 last 69.48) - open/close 22069 (time 1787341893.40 total 1018.14 last 70.64) - open/close 22746 (time 1787341903.41 total 1028.15 last 67.63) - open/close 23443 (time 1787341913.42 total 1038.16 last 69.61) - open/close 24164 (time 1787341923.48 total 1048.22 last 71.63) - open/close 24841 (time 1787341933.49 total 1058.23 last 67.65) - open/close 25552 (time 1787341943.51 total 1068.25 last 70.94) - open/close 26224 (time 1787341953.52 total 1078.26 last 67.16) - open/close 26885 (time 1787341963.52 total 1088.26 last 66.09) - open/close 27563 (time 1787341973.53 total 1098.27 last 67.75) - open/close 28209 (time 1787341983.54 total 1108.28 last 64.53) - open/close 28885 (time 1787341993.55 total 1118.29 last 67.55) - open/close 29533 (time 1787342003.56 total 1128.30 last 64.72) - open/close 30000 (time 1787342010.18 total 1134.92 last 70.58) - open/close 30681 (time 1787342020.20 total 1144.94 last 67.96) - open/close 31346 (time 1787342030.21 total 1154.95 last 66.39) - open/close 32024 (time 1787342040.23 total 1164.97 last 67.72) - open/close 32619 (time 1787342050.24 total 1174.98 last 59.43) - open/close 33221 (time 1787342060.25 total 1184.99 last 60.10) - open/close 33803 (time 1787342070.26 total 1195.00 last 58.18) - open/close 34502 (time 1787342080.27 total 1205.01 last 69.84) - open/close 35168 (time 1787342090.29 total 1215.03 last 66.47) - open/close 35825 (time 1787342100.30 total 1225.04 last 65.60) - open/close 36479 (time 1787342110.31 total 1235.05 last 65.36) - open/close 37069 (time 1787342120.32 total 1245.06 last 58.93) - open/close 37654 (time 1787342130.33 total 1255.07 last 58.44) - open/close 38194 (time 1787342140.33 total 1265.07 last 53.98) - open/close 38793 (time 1787342150.34 total 1275.08 last 59.84) - open/close 39390 (time 1787342160.38 total 1285.12 last 59.50) - open/close 40000 (time 1787342170.25 total 1294.99 last 61.78) - open/close 40647 (time 1787342180.27 total 1305.01 last 64.56) - open/close 41291 (time 1787342190.28 total 1315.02 last 64.36) - open/close 41901 (time 1787342200.28 total 1325.02 last 60.99) - open/close 42416 (time 1787342210.28 total 1335.02 last 51.48) - open/close 42933 (time 1787342220.29 total 1345.03 last 51.67) - open/close 43482 (time 1787342230.29 total 1355.03 last 54.90) - open/close 44058 (time 1787342240.30 total 1365.04 last 57.55) - open/close 44717 (time 1787342250.30 total 1375.04 last 65.88) - open/close 45369 (time 1787342260.30 total 1385.04 last 65.20) - open/close 45979 (time 1787342270.31 total 1395.05 last 60.93) - open/close 46636 (time 1787342280.32 total 1405.06 last 65.63) - open/close 47280 (time 1787342290.34 total 1415.08 last 64.31) - open/close 47865 (time 1787342300.35 total 1425.09 last 58.40) - open/close 48447 (time 1787342310.35 total 1435.09 last 58.19) - open/close 49023 (time 1787342320.37 total 1445.11 last 57.54) - open/close 49533 (time 1787342330.37 total 1455.11 last 50.95) total: 50000 open/close in 1463.20 seconds: 34.17 ops/second - unlinked 0 (time 1787342344 ; total 0 ; last 0) - unlinked 10000 (time 1787342513 ; total 169 ; last 169) - unlinked 20000 (time 1787342675 ; total 331 ; last 162) - unlinked 30000 (time 1787342838 ; total 494 ; last 163) - unlinked 40000 (time 1787342997 ; total 653 ; last 159) total: 50000 unlinks in 822 seconds: 60.827251 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg318-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg318-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg318-server stop mds service on oleg318-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg318-server unloading modules via unload_modules_local on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing unload_modules_local oleg318-server: modules unloaded. pdsh@oleg318-client: oleg318-server: ssh exited with exit code 2 pdsh@oleg318-client: oleg318-client: ssh exited with exit code 2 pdsh@oleg318-client: oleg318-server: ssh exited with exit code 2 pdsh@oleg318-client: oleg318-client: ssh exited with exit code 2