== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 08:08:01 (1786104481) start mds service on oleg421-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg421-server' oleg421-server: oleg421-server.virtnet: executing load_modules_local oleg421-server: Loading modules from /home/green/git/lustre-release/lustre oleg421-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg421-server: detected 4 online CPUs by sysfs oleg421-server: MODOPTS_LIBCFS= oleg421-server: Force libcfs to create 2 CPU partitions oleg421-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg421-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg421-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg421-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-OST0000 oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg421-client.virtnet: -o user_xattr,flock 192.168.204.121@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 84 (time 1786104589.31 total 10.01 last 8.39) - open/close 220 (time 1786104599.63 total 20.33 last 13.18) - open/close 380 (time 1786104610.26 total 30.96 last 15.05) - open/close 571 (time 1786104622.40 total 43.10 last 15.74) - open/close 708 (time 1786104632.41 total 53.11 last 13.69) - open/close 862 (time 1786104642.42 total 63.12 last 15.38) - open/close 966 (time 1786104652.43 total 73.13 last 10.39) - open/close 1033 (time 1786104662.44 total 83.14 last 6.70) - open/close 1185 (time 1786104672.45 total 93.15 last 15.18) - open/close 1464 (time 1786104682.48 total 103.18 last 27.82) - open/close 1610 (time 1786104693.07 total 113.77 last 13.79) - open/close 1670 (time 1786104703.89 total 124.59 last 5.55) - open/close 1721 (time 1786104713.92 total 134.62 last 5.09) - open/close 1817 (time 1786104723.94 total 144.64 last 9.58) - open/close 1949 (time 1786104734.91 total 155.61 last 12.02) - open/close 2082 (time 1786104745.17 total 165.87 last 12.97) - open/close 2229 (time 1786104755.17 total 175.87 last 14.69) - open/close 2327 (time 1786104765.18 total 185.88 last 9.80) - open/close 2524 (time 1786104775.38 total 196.08 last 19.31) - open/close 2673 (time 1786104785.40 total 206.10 last 14.87) - open/close 2790 (time 1786104795.43 total 216.13 last 11.67) - open/close 2904 (time 1786104805.70 total 226.40 last 11.10) - open/close 3005 (time 1786104815.76 total 236.46 last 10.03) - open/close 3093 (time 1786104825.79 total 246.49 last 8.78) - open/close 3223 (time 1786104836.42 total 257.12 last 12.23) - open/close 3378 (time 1786104846.72 total 267.42 last 15.04) - open/close 3469 (time 1786104856.75 total 277.45 last 9.07) - open/close 3595 (time 1786104867.54 total 288.24 last 11.68) - open/close 3728 (time 1786104877.56 total 298.26 last 13.27) - open/close 3847 (time 1786104887.58 total 308.28 last 11.88) - open/close 3971 (time 1786104898.30 total 319.00 last 11.57) - open/close 4114 (time 1786104909.20 total 329.90 last 13.13) - open/close 4287 (time 1786104919.23 total 339.92 last 17.26) - open/close 4411 (time 1786104929.98 total 350.68 last 11.53) - open/close 4517 (time 1786104940.00 total 360.70 last 10.59) - open/close 4664 (time 1786104950.74 total 371.44 last 13.69) - open/close 4899 (time 1786104960.74 total 381.44 last 23.49) - open/close 5044 (time 1786104971.26 total 391.96 last 13.78) - open/close 5206 (time 1786104981.92 total 402.62 last 15.20) - open/close 5322 (time 1786104991.92 total 412.62 last 11.60) - open/close 5492 (time 1786105001.93 total 422.63 last 16.99) - open/close 5634 (time 1786105011.96 total 432.65 last 14.16) - open/close 5732 (time 1786105021.96 total 442.66 last 9.80) - open/close 5906 (time 1786105032.45 total 453.15 last 16.59) - open/close 6037 (time 1786105042.89 total 463.58 last 12.55) - open/close 6151 (time 1786105053.77 total 474.47 last 10.48) - open/close 6240 (time 1786105063.79 total 484.49 last 8.88) - open/close 6393 (time 1786105073.79 total 494.49 last 15.29) - open/close 6480 (time 1786105083.92 total 504.62 last 8.59) - open/close 6618 (time 1786105094.69 total 515.39 last 12.81) - open/close 6746 (time 1786105105.34 total 526.03 last 12.03) - open/close 6881 (time 1786105118.34 total 539.04 last 10.38) - open/close 7013 (time 1786105128.70 total 549.40 last 12.74) - open/close 7163 (time 1786105139.60 total 560.29 last 13.77) - open/close 7311 (time 1786105149.60 total 570.30 last 14.79) - open/close 7448 (time 1786105159.61 total 580.30 last 13.70) - open/close 7534 (time 1786105169.62 total 590.31 last 8.59) - open/close 7641 (time 1786105179.65 total 600.35 last 10.67) - open/close 7793 (time 1786105190.47 total 611.17 last 14.05) - open/close 7886 (time 1786105200.59 total 621.29 last 9.19) - open/close 8038 (time 1786105210.64 total 631.33 last 15.13) - open/close 8099 (time 1786105220.81 total 641.51 last 6.00) - open/close 8224 (time 1786105231.65 total 652.35 last 11.53) - open/close 8335 (time 1786105242.65 total 663.35 last 10.09) - open/close 8443 (time 1786105253.00 total 673.70 last 10.43) - open/close 8576 (time 1786105263.02 total 683.72 last 13.28) - open/close 8808 (time 1786105276.00 total 696.70 last 17.87) - open/close 9014 (time 1786105286.03 total 706.73 last 20.54) - open/close 9128 (time 1786105296.60 total 717.30 last 10.79) - open/close 9271 (time 1786105306.94 total 727.64 last 13.83) - open/close 9380 (time 1786105317.01 total 737.71 last 10.82) - open/close 9521 (time 1786105327.83 total 748.53 last 13.04) - open/close 9602 (time 1786105338.18 total 758.88 last 7.83) - open/close 9748 (time 1786105348.61 total 769.31 last 14.00) - open/close 9893 (time 1786105358.62 total 779.32 last 14.48) - open/close 10000 (time 1786105367.10 total 787.80 last 12.62) - open/close 10156 (time 1786105377.98 total 798.68 last 14.34) - open/close 10313 (time 1786105388.12 total 808.82 last 15.48) - open/close 10406 (time 1786105400.23 total 820.93 last 7.68) - open/close 10527 (time 1786105410.23 total 830.93 last 12.10) - open/close 10719 (time 1786105420.25 total 840.95 last 19.17) - open/close 10810 (time 1786105431.05 total 851.74 last 8.43) - open/close 10993 (time 1786105441.50 total 862.20 last 17.50) - open/close 11200 (time 1786105452.00 total 872.70 last 19.71) - open/close 11277 (time 1786105462.44 total 883.14 last 7.38) - open/close 11425 (time 1786105472.45 total 893.15 last 14.78) - open/close 11621 (time 1786105483.24 total 903.94 last 18.17) - open/close 11740 (time 1786105493.34 total 914.04 last 11.78) - open/close 11971 (time 1786105503.36 total 924.06 last 23.04) - open/close 12096 (time 1786105514.27 total 934.96 last 11.46) - open/close 12315 (time 1786105525.25 total 945.95 last 19.93) - open/close 12427 (time 1786105535.29 total 955.99 last 11.16) - open/close 12633 (time 1786105545.29 total 965.99 last 20.59) - open/close 12759 (time 1786105558.35 total 979.05 last 9.65) - open/close 12889 (time 1786105568.79 total 989.49 last 12.45) - open/close 13141 (time 1786105578.80 total 999.49 last 25.20) - open/close 13334 (time 1786105588.87 total 1009.57 last 19.16) - open/close 13539 (time 1786105598.95 total 1019.65 last 20.33) - open/close 13725 (time 1786105609.85 total 1030.55 last 17.07) - open/close 13863 (time 1786105620.30 total 1041.00 last 13.20) - open/close 14063 (time 1786105631.33 total 1052.03 last 18.13) - open/close 14209 (time 1786105641.35 total 1062.05 last 14.57) - open/close 14317 (time 1786105651.37 total 1072.07 last 10.78) - open/close 14462 (time 1786105662.21 total 1082.91 last 13.38) - open/close 14529 (time 1786105672.21 total 1092.91 last 6.70) - open/close 14660 (time 1786105682.70 total 1103.40 last 12.49) - open/close 14801 (time 1786105692.71 total 1113.41 last 14.08) - open/close 14908 (time 1786105705.24 total 1125.94 last 8.54) - open/close 15155 (time 1786105715.46 total 1136.16 last 24.17) - open/close 15277 (time 1786105725.70 total 1146.40 last 11.91) - open/close 15447 (time 1786105736.08 total 1156.78 last 16.38) - open/close 15549 (time 1786105746.71 total 1167.41 last 9.59) - open/close 15732 (time 1786105757.46 total 1178.16 last 17.02) - open/close 15967 (time 1786105767.47 total 1188.17 last 23.48) - open/close 16051 (time 1786105778.08 total 1198.78 last 7.92) - open/close 16150 (time 1786105788.11 total 1208.81 last 9.87) - open/close 16553 (time 1786105798.13 total 1218.83 last 40.22) - open/close 17223 (time 1786105808.16 total 1228.86 last 66.77) - open/close 17901 (time 1786105818.17 total 1238.86 last 67.77) - open/close 18567 (time 1786105828.17 total 1248.87 last 66.57) - open/close 19282 (time 1786105838.18 total 1258.88 last 71.44) - open/close 19880 (time 1786105848.19 total 1268.89 last 59.74) - open/close 20000 (time 1786105850.23 total 1270.93 last 58.87) - open/close 20700 (time 1786105860.24 total 1280.93 last 69.94) - open/close 21368 (time 1786105870.24 total 1290.94 last 66.77) - open/close 22022 (time 1786105880.26 total 1300.96 last 65.24) - open/close 22709 (time 1786105890.27 total 1310.97 last 68.68) - open/close 23411 (time 1786105900.28 total 1320.97 last 70.14) - open/close 24108 (time 1786105910.29 total 1330.99 last 69.63) - open/close 24794 (time 1786105920.30 total 1341.00 last 68.52) - open/close 25450 (time 1786105930.30 total 1351.00 last 65.57) - open/close 26137 (time 1786105940.31 total 1361.01 last 68.63) - open/close 26872 (time 1786105950.33 total 1371.03 last 73.39) - open/close 27541 (time 1786105960.34 total 1381.04 last 66.82) - open/close 28201 (time 1786105970.34 total 1391.04 last 65.98) - open/close 28916 (time 1786105980.36 total 1401.06 last 71.40) - open/close 29609 (time 1786105990.36 total 1411.06 last 69.29) - open/close 30000 (time 1786105996.06 total 1416.76 last 68.56) - open/close 30661 (time 1786106006.08 total 1426.78 last 65.99) - open/close 31303 (time 1786106016.10 total 1436.80 last 64.08) - open/close 31969 (time 1786106026.10 total 1446.80 last 66.58) - open/close 32646 (time 1786106036.11 total 1456.81 last 67.63) - open/close 33307 (time 1786106046.11 total 1466.81 last 66.10) - open/close 34000 (time 1786106056.13 total 1476.83 last 69.16) - open/close 34674 (time 1786106066.14 total 1486.84 last 67.34) - open/close 35360 (time 1786106076.15 total 1496.84 last 68.56) - open/close 36055 (time 1786106086.15 total 1506.85 last 69.47) - open/close 36754 (time 1786106096.16 total 1516.86 last 69.85) - open/close 37393 (time 1786106106.19 total 1526.89 last 63.68) - open/close 38031 (time 1786106116.20 total 1536.90 last 63.72) - open/close 38708 (time 1786106126.25 total 1546.95 last 67.40) - open/close 39395 (time 1786106136.26 total 1556.96 last 68.62) - open/close 40000 (time 1786106145.49 total 1566.19 last 65.53) - open/close 40710 (time 1786106155.50 total 1576.20 last 70.97) - open/close 41380 (time 1786106165.51 total 1586.21 last 66.88) - open/close 42027 (time 1786106175.52 total 1596.22 last 64.64) - open/close 42688 (time 1786106185.52 total 1606.22 last 66.09) - open/close 43320 (time 1786106195.53 total 1616.23 last 63.16) - open/close 44000 (time 1786106205.55 total 1626.25 last 67.88) - open/close 44683 (time 1786106215.55 total 1636.25 last 68.28) - open/close 45327 (time 1786106225.56 total 1646.26 last 64.33) - open/close 45962 (time 1786106235.58 total 1656.28 last 63.36) - open/close 46555 (time 1786106245.58 total 1666.28 last 59.30) - open/close 47183 (time 1786106255.59 total 1676.29 last 62.77) - open/close 47788 (time 1786106265.60 total 1686.30 last 60.43) - open/close 48457 (time 1786106275.61 total 1696.31 last 66.81) - open/close 49114 (time 1786106285.62 total 1706.32 last 65.67) - open/close 49784 (time 1786106295.62 total 1716.32 last 66.96) total: 50000 open/close in 1719.90 seconds: 29.07 ops/second - unlinked 0 (time 1786106304 ; total 0 ; last 0) - unlinked 10000 (time 1786106472 ; total 168 ; last 168) - unlinked 20000 (time 1786106649 ; total 345 ; last 177) - unlinked 30000 (time 1786106831 ; total 527 ; last 182) - unlinked 40000 (time 1786107018 ; total 714 ; last 187) total: 50000 unlinks in 887 seconds: 56.369785 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg421-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg421-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg421-server stop mds service on oleg421-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg421-server unloading modules via unload_modules_local on: 'oleg421-server' oleg421-server: oleg421-server.virtnet: executing unload_modules_local oleg421-server: modules unloaded. pdsh@oleg421-client: oleg421-client: ssh exited with exit code 2 pdsh@oleg421-client: oleg421-server: ssh exited with exit code 2 pdsh@oleg421-client: oleg421-server: ssh exited with exit code 2 pdsh@oleg421-client: oleg421-client: ssh exited with exit code 2