== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:47:03 (1787464023) start mds service on oleg140-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg140-server' oleg140-server: oleg140-server.virtnet: executing load_modules_local oleg140-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg140-server: Loading modules from /home/green/git/lustre-release/lustre oleg140-server: detected 4 online CPUs by sysfs oleg140-server: MODOPTS_LIBCFS= oleg140-server: Force libcfs to create 2 CPU partitions oleg140-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg140-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg140-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg140-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-OST0000 oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg140-client.virtnet: -o user_xattr,flock 192.168.201.140@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 166 (time 1787464108.90 total 10.22 last 16.25) - open/close 223 (time 1787464119.81 total 21.13 last 5.22) - open/close 338 (time 1787464130.33 total 31.64 last 10.94) - open/close 507 (time 1787464140.34 total 41.66 last 16.87) - open/close 703 (time 1787464150.36 total 51.68 last 19.57) - open/close 958 (time 1787464160.52 total 61.84 last 25.10) - open/close 1200 (time 1787464170.55 total 71.86 last 24.13) - open/close 1451 (time 1787464180.58 total 81.89 last 25.02) - open/close 1602 (time 1787464192.46 total 93.78 last 12.71) - open/close 1841 (time 1787464202.51 total 103.83 last 23.77) - open/close 1945 (time 1787464212.52 total 113.84 last 10.39) - open/close 2184 (time 1787464223.53 total 124.85 last 21.70) - open/close 2372 (time 1787464234.53 total 135.84 last 17.10) - open/close 2496 (time 1787464245.19 total 146.51 last 11.62) - open/close 2596 (time 1787464255.27 total 156.59 last 9.93) - open/close 2725 (time 1787464265.94 total 167.26 last 12.08) - open/close 2883 (time 1787464276.69 total 178.01 last 14.70) - open/close 3054 (time 1787464286.74 total 188.05 last 17.03) - open/close 3193 (time 1787464297.03 total 198.35 last 13.50) - open/close 3269 (time 1787464307.67 total 208.99 last 7.14) - open/close 3365 (time 1787464317.73 total 219.05 last 9.55) - open/close 3490 (time 1787464327.75 total 229.06 last 12.48) - open/close 3629 (time 1787464338.61 total 239.93 last 12.80) - open/close 3739 (time 1787464349.28 total 250.60 last 10.31) - open/close 3844 (time 1787464360.03 total 261.35 last 9.77) - open/close 3988 (time 1787464371.13 total 272.44 last 12.98) - open/close 4125 (time 1787464381.77 total 283.09 last 12.87) - open/close 4364 (time 1787464391.78 total 293.10 last 23.88) - open/close 4584 (time 1787464402.18 total 303.50 last 21.15) - open/close 4807 (time 1787464412.19 total 313.51 last 22.28) - open/close 4993 (time 1787464422.64 total 323.96 last 17.80) - open/close 5058 (time 1787464432.68 total 334.00 last 6.47) - open/close 5220 (time 1787464442.92 total 344.24 last 15.81) - open/close 5366 (time 1787464453.45 total 354.77 last 13.87) - open/close 5432 (time 1787464464.80 total 366.12 last 5.82) - open/close 5528 (time 1787464474.86 total 376.17 last 9.55) - open/close 5652 (time 1787464485.33 total 386.65 last 11.84) - open/close 5813 (time 1787464495.70 total 397.02 last 15.53) - open/close 5920 (time 1787464506.44 total 407.76 last 9.96) - open/close 6091 (time 1787464516.94 total 418.26 last 16.29) - open/close 6246 (time 1787464526.96 total 428.27 last 15.48) - open/close 6386 (time 1787464536.96 total 438.28 last 14.00) - open/close 6476 (time 1787464549.98 total 451.30 last 6.91) - open/close 6572 (time 1787464560.02 total 461.34 last 9.56) - open/close 6738 (time 1787464570.05 total 471.37 last 16.55) - open/close 6829 (time 1787464583.12 total 484.44 last 6.96) - open/close 6921 (time 1787464593.76 total 495.08 last 8.64) - open/close 7073 (time 1787464604.64 total 505.95 last 13.98) - open/close 7179 (time 1787464614.67 total 515.99 last 10.57) - open/close 7335 (time 1787464624.69 total 526.00 last 15.57) - open/close 7451 (time 1787464634.83 total 536.15 last 11.44) - open/close 7544 (time 1787464647.38 total 548.70 last 7.41) - open/close 7647 (time 1787464657.38 total 558.70 last 10.30) - open/close 7809 (time 1787464667.38 total 568.70 last 16.20) - open/close 7914 (time 1787464678.18 total 579.50 last 9.73) - open/close 8039 (time 1787464688.35 total 589.67 last 12.29) - open/close 8116 (time 1787464698.88 total 600.19 last 7.31) - open/close 8180 (time 1787464709.15 total 610.47 last 6.23) - open/close 8262 (time 1787464721.67 total 622.99 last 6.55) - open/close 8384 (time 1787464732.07 total 633.39 last 11.73) - open/close 8463 (time 1787464743.00 total 644.31 last 7.23) - open/close 8560 (time 1787464753.69 total 655.01 last 9.07) - open/close 8653 (time 1787464763.73 total 665.05 last 9.26) - open/close 8717 (time 1787464774.74 total 676.06 last 5.82) - open/close 8803 (time 1787464785.31 total 686.63 last 8.14) - open/close 8902 (time 1787464795.98 total 697.30 last 9.28) - open/close 9000 (time 1787464806.02 total 707.34 last 9.77) - open/close 9120 (time 1787464816.03 total 717.34 last 11.99) - open/close 9222 (time 1787464826.90 total 728.22 last 9.38) - open/close 9401 (time 1787464837.10 total 738.41 last 17.56) - open/close 9497 (time 1787464847.76 total 749.08 last 9.00) - open/close 9575 (time 1787464857.77 total 759.08 last 7.80) - open/close 9675 (time 1787464867.77 total 769.09 last 9.99) - open/close 9804 (time 1787464878.22 total 779.54 last 12.34) - open/close 9855 (time 1787464889.00 total 790.32 last 4.73) - open/close 10000 (time 1787464898.73 total 800.04 last 14.91) - open/close 10144 (time 1787464909.28 total 810.59 last 13.65) - open/close 10344 (time 1787464925.15 total 826.47 last 12.60) - open/close 10499 (time 1787464935.82 total 837.14 last 14.53) - open/close 10606 (time 1787464946.95 total 848.27 last 9.61) - open/close 10688 (time 1787464957.26 total 858.58 last 7.96) - open/close 10844 (time 1787464967.97 total 869.29 last 14.56) - open/close 11047 (time 1787464977.97 total 879.29 last 20.30) - open/close 11138 (time 1787464987.99 total 889.31 last 9.08) - open/close 11235 (time 1787464998.01 total 899.33 last 9.68) - open/close 11351 (time 1787465010.24 total 911.56 last 9.49) - open/close 11504 (time 1787465020.75 total 922.07 last 14.55) - open/close 11611 (time 1787465030.76 total 932.07 last 10.70) - open/close 11849 (time 1787465041.30 total 942.61 last 22.58) - open/close 11964 (time 1787465052.27 total 953.59 last 10.48) - open/close 12137 (time 1787465062.28 total 963.60 last 17.29) - open/close 12305 (time 1787465073.18 total 974.50 last 15.41) - open/close 12384 (time 1787465084.75 total 986.07 last 6.82) - open/close 12484 (time 1787465094.76 total 996.07 last 10.00) - open/close 12554 (time 1787465104.89 total 1006.21 last 6.91) - open/close 12650 (time 1787465115.41 total 1016.73 last 9.12) - open/close 12712 (time 1787465125.51 total 1026.83 last 6.14) - open/close 12841 (time 1787465135.63 total 1036.95 last 12.75) - open/close 12892 (time 1787465148.49 total 1049.81 last 3.96) - open/close 13035 (time 1787465158.53 total 1059.85 last 14.25) - open/close 13132 (time 1787465169.29 total 1070.61 last 9.02) - open/close 13285 (time 1787465181.39 total 1082.70 last 12.65) - open/close 13420 (time 1787465191.42 total 1092.73 last 13.46) - open/close 13552 (time 1787465202.46 total 1103.77 last 11.96) - open/close 13733 (time 1787465213.50 total 1114.82 last 16.38) - open/close 13888 (time 1787465224.14 total 1125.45 last 14.58) - open/close 14019 (time 1787465235.03 total 1136.35 last 12.03) - open/close 14114 (time 1787465245.40 total 1146.72 last 9.16) - open/close 14205 (time 1787465255.82 total 1157.13 last 8.73) - open/close 14655 (time 1787465265.82 total 1167.14 last 44.99) - open/close 15315 (time 1787465275.83 total 1177.15 last 65.93) - open/close 15905 (time 1787465285.84 total 1187.15 last 58.97) - open/close 16571 (time 1787465295.84 total 1197.16 last 66.54) - open/close 17236 (time 1787465305.85 total 1207.16 last 66.48) - open/close 17922 (time 1787465315.86 total 1217.18 last 68.51) - open/close 18663 (time 1787465325.86 total 1227.18 last 74.10) - open/close 19310 (time 1787465335.86 total 1237.18 last 64.68) - open/close 19998 (time 1787465345.87 total 1247.19 last 68.72) - open/close 20798 (time 1787465355.88 total 1257.20 last 79.94) - open/close 21508 (time 1787465365.89 total 1267.21 last 70.95) - open/close 22246 (time 1787465375.90 total 1277.22 last 73.71) - open/close 22974 (time 1787465385.90 total 1287.22 last 72.79) - open/close 23587 (time 1787465395.92 total 1297.24 last 61.21) - open/close 24312 (time 1787465405.93 total 1307.25 last 72.43) - open/close 24947 (time 1787465415.94 total 1317.26 last 63.44) - open/close 25548 (time 1787465425.94 total 1327.26 last 60.10) - open/close 26232 (time 1787465435.94 total 1337.26 last 68.37) - open/close 27012 (time 1787465445.95 total 1347.26 last 77.98) - open/close 27659 (time 1787465455.96 total 1357.28 last 64.59) - open/close 28277 (time 1787465465.97 total 1367.29 last 61.77) - open/close 28961 (time 1787465475.98 total 1377.30 last 68.33) - open/close 29452 (time 1787465486.00 total 1387.32 last 48.98) - open/close 30000 (time 1787465494.62 total 1395.94 last 63.60) - open/close 30611 (time 1787465504.63 total 1405.95 last 61.05) - open/close 31206 (time 1787465514.65 total 1415.97 last 59.34) - open/close 31781 (time 1787465524.67 total 1425.99 last 57.41) - open/close 32193 (time 1787465534.68 total 1436.00 last 41.15) - open/close 32607 (time 1787465544.69 total 1446.01 last 41.37) - open/close 33309 (time 1787465554.71 total 1456.03 last 70.07) - open/close 33982 (time 1787465564.71 total 1466.03 last 67.25) - open/close 34428 (time 1787465574.73 total 1476.04 last 44.55) - open/close 34867 (time 1787465584.74 total 1486.06 last 43.82) - open/close 35491 (time 1787465594.76 total 1496.08 last 62.29) - open/close 36016 (time 1787465604.80 total 1506.11 last 52.33) - open/close 36663 (time 1787465614.81 total 1516.13 last 64.61) - open/close 37390 (time 1787465624.82 total 1526.13 last 72.65) - open/close 38120 (time 1787465634.82 total 1536.14 last 72.96) - open/close 38781 (time 1787465644.84 total 1546.16 last 65.96) - open/close 39370 (time 1787465654.86 total 1556.17 last 58.82) - open/close 39926 (time 1787465664.87 total 1566.19 last 55.52) - open/close 40597 (time 1787465674.88 total 1576.20 last 67.02) - open/close 41260 (time 1787465684.89 total 1586.20 last 66.28) - open/close 42080 (time 1787465694.90 total 1596.21 last 81.92) - open/close 42840 (time 1787465704.91 total 1606.23 last 75.89) - open/close 43441 (time 1787465714.92 total 1616.24 last 60.03) - open/close 44293 (time 1787465724.93 total 1626.25 last 85.15) - open/close 45030 (time 1787465734.93 total 1636.25 last 73.69) - open/close 45639 (time 1787465744.93 total 1646.25 last 60.89) - open/close 46338 (time 1787465754.96 total 1656.28 last 69.72) - open/close 47066 (time 1787465764.98 total 1666.30 last 72.66) - open/close 47794 (time 1787465775.01 total 1676.32 last 72.59) - open/close 48508 (time 1787465785.01 total 1686.33 last 71.36) - open/close 49177 (time 1787465795.02 total 1696.34 last 66.83) - open/close 49765 (time 1787465805.04 total 1706.36 last 58.71) total: 50000 open/close in 1709.91 seconds: 29.24 ops/second - unlinked 0 (time 1787465814 ; total 0 ; last 0) - unlinked 10000 (time 1787465958 ; total 144 ; last 144) - unlinked 20000 (time 1787466126 ; total 312 ; last 168) - unlinked 30000 (time 1787466287 ; total 473 ; last 161) - unlinked 40000 (time 1787466448 ; total 634 ; last 161) total: 50000 unlinks in 786 seconds: 63.613232 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg140-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg140-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg140-server stop mds service on oleg140-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg140-server unloading modules via unload_modules_local on: 'oleg140-server' oleg140-server: oleg140-server.virtnet: executing unload_modules_local oleg140-server: modules unloaded. pdsh@oleg140-client: oleg140-client: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-server: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-client: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-server: ssh exited with exit code 2