== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 19:05:55 (1786403155) start mds service on oleg349-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg349-server' oleg349-server: oleg349-server.virtnet: executing load_modules_local oleg349-server: Loading modules from /home/green/git/lustre-release/lustre oleg349-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg349-server: detected 4 online CPUs by sysfs oleg349-server: MODOPTS_LIBCFS= oleg349-server: Force libcfs to create 2 CPU partitions oleg349-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg349-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg349-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg349-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-OST0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg349-client.virtnet: -o user_xattr,flock 192.168.203.149@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 130 (time 1786403243.38 total 10.91 last 11.92) - open/close 227 (time 1786403254.21 total 21.74 last 8.95) - open/close 434 (time 1786403266.62 total 34.15 last 16.68) - open/close 655 (time 1786403276.79 total 44.32 last 21.74) - open/close 865 (time 1786403287.31 total 54.84 last 19.96) - open/close 1049 (time 1786403297.55 total 65.08 last 17.96) - open/close 1212 (time 1786403307.56 total 75.09 last 16.29) - open/close 1423 (time 1786403317.93 total 85.46 last 20.35) - open/close 1621 (time 1786403327.99 total 95.52 last 19.67) - open/close 1801 (time 1786403338.00 total 105.53 last 17.99) - open/close 1995 (time 1786403348.02 total 115.55 last 19.36) - open/close 2137 (time 1786403358.04 total 125.57 last 14.17) - open/close 2293 (time 1786403368.65 total 136.18 last 14.70) - open/close 2466 (time 1786403379.66 total 147.19 last 15.72) - open/close 2619 (time 1786403389.72 total 157.25 last 15.21) - open/close 2765 (time 1786403399.92 total 167.45 last 14.31) - open/close 2940 (time 1786403409.92 total 177.46 last 17.49) - open/close 3089 (time 1786403419.93 total 187.46 last 14.90) - open/close 3212 (time 1786403429.95 total 197.48 last 12.28) - open/close 3387 (time 1786403439.98 total 207.51 last 17.44) - open/close 3576 (time 1786403450.01 total 217.54 last 18.85) - open/close 3779 (time 1786403460.02 total 227.55 last 20.28) - open/close 3943 (time 1786403470.22 total 237.75 last 16.07) - open/close 4109 (time 1786403480.70 total 248.23 last 15.84) - open/close 4314 (time 1786403490.71 total 258.24 last 20.48) - open/close 4503 (time 1786403500.71 total 268.24 last 18.90) - open/close 4612 (time 1786403510.73 total 278.26 last 10.88) - open/close 4816 (time 1786403520.84 total 288.37 last 20.17) - open/close 4903 (time 1786403532.49 total 300.02 last 7.47) - open/close 5092 (time 1786403543.60 total 311.13 last 17.02) - open/close 5208 (time 1786403556.43 total 323.96 last 9.04) - open/close 5374 (time 1786403566.49 total 334.03 last 16.49) - open/close 5569 (time 1786403576.51 total 344.04 last 19.47) - open/close 5775 (time 1786403586.51 total 354.04 last 20.59) - open/close 5983 (time 1786403597.22 total 364.75 last 19.42) - open/close 6183 (time 1786403608.21 total 375.74 last 18.20) - open/close 6415 (time 1786403618.77 total 386.30 last 21.98) - open/close 6648 (time 1786403629.04 total 396.57 last 22.69) - open/close 6837 (time 1786403639.05 total 406.58 last 18.88) - open/close 7137 (time 1786403650.14 total 417.67 last 27.03) - open/close 7262 (time 1786403660.87 total 428.41 last 11.65) - open/close 7493 (time 1786403670.94 total 438.47 last 22.96) - open/close 7656 (time 1786403680.95 total 448.48 last 16.27) - open/close 7864 (time 1786403691.79 total 459.32 last 19.19) - open/close 8038 (time 1786403702.09 total 469.62 last 16.90) - open/close 8312 (time 1786403712.52 total 480.05 last 26.27) - open/close 8519 (time 1786403722.52 total 490.05 last 20.69) - open/close 8704 (time 1786403733.58 total 501.11 last 16.74) - open/close 8903 (time 1786403743.84 total 511.37 last 19.39) - open/close 9052 (time 1786403753.84 total 521.37 last 14.90) - open/close 9145 (time 1786403764.94 total 532.47 last 8.38) - open/close 9367 (time 1786403774.94 total 542.47 last 22.20) - open/close 9495 (time 1786403785.00 total 552.53 last 12.72) - open/close 9639 (time 1786403795.07 total 562.60 last 14.30) - open/close 9819 (time 1786403805.73 total 573.26 last 16.88) - open/close 9963 (time 1786403817.17 total 584.70 last 12.58) - open/close 10166 (time 1786403827.53 total 595.06 last 19.60) - open/close 10307 (time 1786403837.55 total 605.08 last 14.07) - open/close 10487 (time 1786403848.47 total 616.01 last 16.48) - open/close 10753 (time 1786403858.48 total 626.01 last 26.59) - open/close 10938 (time 1786403868.50 total 636.04 last 18.45) - open/close 11152 (time 1786403878.51 total 646.04 last 21.39) - open/close 11326 (time 1786403889.52 total 657.05 last 15.81) - open/close 11543 (time 1786403899.54 total 667.07 last 21.66) - open/close 11786 (time 1786403909.55 total 677.08 last 24.28) - open/close 11933 (time 1786403919.76 total 687.29 last 14.40) - open/close 12130 (time 1786403930.03 total 697.56 last 19.17) - open/close 12366 (time 1786403940.71 total 708.25 last 22.09) - open/close 12530 (time 1786403950.75 total 718.29 last 16.33) - open/close 12716 (time 1786403960.87 total 728.40 last 18.39) - open/close 12778 (time 1786403970.91 total 738.45 last 6.17) - open/close 12993 (time 1786403980.95 total 748.48 last 21.43) - open/close 13192 (time 1786403991.16 total 758.69 last 19.49) - open/close 13302 (time 1786404002.12 total 769.65 last 10.04) - open/close 13410 (time 1786404013.12 total 780.66 last 9.81) - open/close 13578 (time 1786404023.13 total 790.66 last 16.79) - open/close 13762 (time 1786404033.14 total 800.67 last 18.39) - open/close 13889 (time 1786404043.41 total 810.94 last 12.36) - open/close 14043 (time 1786404053.55 total 821.08 last 15.19) - open/close 14195 (time 1786404063.84 total 831.37 last 14.77) - open/close 14367 (time 1786404073.88 total 841.41 last 17.13) - open/close 14595 (time 1786404083.90 total 851.43 last 22.76) - open/close 14804 (time 1786404094.95 total 862.48 last 18.91) - open/close 14914 (time 1786404105.44 total 872.98 last 10.48) - open/close 15101 (time 1786404116.34 total 883.87 last 17.17) - open/close 15323 (time 1786404126.53 total 894.06 last 21.77) - open/close 15482 (time 1786404136.57 total 904.10 last 15.84) - open/close 15686 (time 1786404147.77 total 915.30 last 18.22) - open/close 15929 (time 1786404157.77 total 925.30 last 24.29) - open/close 16206 (time 1786404167.84 total 935.37 last 27.51) - open/close 16382 (time 1786404177.94 total 945.47 last 17.42) - open/close 16530 (time 1786404187.97 total 955.50 last 14.75) - open/close 16773 (time 1786404197.97 total 965.50 last 24.30) - open/close 16898 (time 1786404208.12 total 975.65 last 12.32) - open/close 17076 (time 1786404218.64 total 986.17 last 16.91) - open/close 17289 (time 1786404229.35 total 996.88 last 19.90) - open/close 17484 (time 1786404239.39 total 1006.92 last 19.41) - open/close 17681 (time 1786404249.80 total 1017.33 last 18.92) - open/close 17933 (time 1786404259.80 total 1027.34 last 25.19) - open/close 18120 (time 1786404269.86 total 1037.39 last 18.60) - open/close 18283 (time 1786404280.28 total 1047.81 last 15.64) - open/close 18492 (time 1786404290.29 total 1057.82 last 20.87) - open/close 18692 (time 1786404300.84 total 1068.37 last 18.96) - open/close 18894 (time 1786404310.87 total 1078.40 last 20.14) - open/close 19094 (time 1786404320.88 total 1088.41 last 19.99) - open/close 19856 (time 1786404330.89 total 1098.42 last 76.08) - open/close 20554 (time 1786404340.89 total 1108.42 last 69.79) - open/close 21286 (time 1786404350.91 total 1118.44 last 73.09) - open/close 22062 (time 1786404360.91 total 1128.44 last 77.57) - open/close 22742 (time 1786404370.99 total 1138.52 last 67.45) - open/close 23368 (time 1786404381.02 total 1148.56 last 62.40) - open/close 24027 (time 1786404391.07 total 1158.60 last 65.59) - open/close 24643 (time 1786404401.08 total 1168.61 last 61.53) - open/close 25380 (time 1786404411.09 total 1178.62 last 73.64) - open/close 26056 (time 1786404421.09 total 1188.63 last 67.58) - open/close 26660 (time 1786404431.10 total 1198.63 last 60.38) - open/close 27324 (time 1786404441.10 total 1208.63 last 66.36) - open/close 28013 (time 1786404451.11 total 1218.64 last 68.88) - open/close 28671 (time 1786404461.14 total 1228.67 last 65.58) - open/close 29368 (time 1786404471.15 total 1238.68 last 69.60) - open/close 30000 (time 1786404479.64 total 1247.17 last 74.50) - open/close 30640 (time 1786404489.64 total 1257.17 last 63.98) - open/close 31307 (time 1786404499.64 total 1267.17 last 66.68) - open/close 32037 (time 1786404509.64 total 1277.17 last 73.00) - open/close 32612 (time 1786404519.66 total 1287.19 last 57.41) - open/close 33249 (time 1786404529.67 total 1297.20 last 63.64) - open/close 34036 (time 1786404539.68 total 1307.21 last 78.60) - open/close 34742 (time 1786404549.69 total 1317.22 last 70.54) - open/close 35395 (time 1786404559.69 total 1327.22 last 65.28) - open/close 36010 (time 1786404569.69 total 1337.23 last 61.48) - open/close 36641 (time 1786404579.71 total 1347.24 last 62.99) - open/close 37308 (time 1786404589.72 total 1357.25 last 66.65) - open/close 38062 (time 1786404599.72 total 1367.25 last 75.37) - open/close 38764 (time 1786404609.73 total 1377.26 last 70.13) - open/close 39418 (time 1786404619.75 total 1387.28 last 65.30) - open/close 40000 (time 1786404628.32 total 1395.85 last 67.93) - open/close 40725 (time 1786404638.32 total 1405.85 last 72.45) - open/close 41401 (time 1786404648.32 total 1415.86 last 67.58) - open/close 42123 (time 1786404658.33 total 1425.86 last 72.15) - open/close 42837 (time 1786404668.33 total 1435.87 last 71.38) - open/close 43493 (time 1786404678.34 total 1445.87 last 65.54) - open/close 44116 (time 1786404688.40 total 1455.93 last 61.94) - open/close 44791 (time 1786404698.41 total 1465.94 last 67.46) - open/close 45390 (time 1786404708.41 total 1475.94 last 59.89) - open/close 46064 (time 1786404718.42 total 1485.95 last 67.32) - open/close 46731 (time 1786404728.42 total 1495.95 last 66.69) - open/close 47370 (time 1786404738.43 total 1505.96 last 63.87) - open/close 48105 (time 1786404748.45 total 1515.98 last 73.33) - open/close 48845 (time 1786404758.46 total 1525.99 last 73.93) - open/close 49471 (time 1786404768.47 total 1536.00 last 62.57) total: 50000 open/close in 1542.93 seconds: 32.41 ops/second - unlinked 0 (time 1786404780 ; total 1 ; last 1) - unlinked 10000 (time 1786404942 ; total 163 ; last 162) - unlinked 20000 (time 1786405107 ; total 328 ; last 165) - unlinked 30000 (time 1786405268 ; total 489 ; last 161) - unlinked 40000 (time 1786405433 ; total 654 ; last 165) total: 50000 unlinks in 818 seconds: 61.124695 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg349-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg349-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg349-server stop mds service on oleg349-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg349-server unloading modules via unload_modules_local on: 'oleg349-server' oleg349-server: oleg349-server.virtnet: executing unload_modules_local oleg349-server: modules unloaded. pdsh@oleg349-client: oleg349-client: ssh exited with exit code 2 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 2 pdsh@oleg349-client: oleg349-client: ssh exited with exit code 2 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 2