== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:01:46 (1787839306) start mds service on oleg619-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg619-server' oleg619-server: oleg619-server.virtnet: executing load_modules_local oleg619-server: Loading modules from /home/green/git/lustre-release/lustre oleg619-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg619-server: detected 4 online CPUs by sysfs oleg619-server: MODOPTS_LIBCFS= oleg619-server: Force libcfs to create 2 CPU partitions oleg619-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg619-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg619-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg619-server: oleg619-server.virtnet: executing set_default_debug -1 all pdsh@oleg619-client: oleg619-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg619-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg619-server: oleg619-server.virtnet: executing set_default_debug -1 all pdsh@oleg619-client: oleg619-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg619-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg619-server: oleg619-server.virtnet: executing set_default_debug -1 all pdsh@oleg619-client: oleg619-server: ssh exited with exit code 1 Started lustre-OST0000 oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg619-client.virtnet: -o user_xattr,flock 192.168.206.119@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 267 (time 1787839412.44 total 10.57 last 25.27) - open/close 538 (time 1787839422.87 total 21.00 last 25.97) - open/close 811 (time 1787839432.87 total 31.00 last 27.29) - open/close 1056 (time 1787839442.87 total 41.00 last 24.50) - open/close 1307 (time 1787839452.89 total 51.02 last 25.06) - open/close 1568 (time 1787839463.19 total 61.32 last 25.35) - open/close 1786 (time 1787839474.21 total 72.34 last 19.78) - open/close 1949 (time 1787839484.21 total 82.34 last 16.29) - open/close 2171 (time 1787839494.36 total 92.49 last 21.88) - open/close 2420 (time 1787839504.37 total 102.50 last 24.86) - open/close 2704 (time 1787839515.13 total 113.26 last 26.41) - open/close 2964 (time 1787839525.59 total 123.72 last 24.84) - open/close 3211 (time 1787839535.59 total 133.72 last 24.70) - open/close 3436 (time 1787839545.60 total 143.73 last 22.48) - open/close 3700 (time 1787839556.62 total 154.75 last 23.97) - open/close 3862 (time 1787839566.63 total 164.76 last 16.18) - open/close 4128 (time 1787839576.63 total 174.76 last 26.60) - open/close 4309 (time 1787839587.47 total 185.60 last 16.70) - open/close 4522 (time 1787839597.65 total 195.78 last 20.93) - open/close 4715 (time 1787839608.22 total 206.35 last 18.26) - open/close 4907 (time 1787839618.93 total 217.06 last 17.93) - open/close 5173 (time 1787839629.21 total 227.34 last 25.86) - open/close 5417 (time 1787839639.21 total 237.34 last 24.40) - open/close 5685 (time 1787839649.31 total 247.44 last 26.55) - open/close 5924 (time 1787839659.41 total 257.54 last 23.67) - open/close 6135 (time 1787839669.97 total 268.10 last 19.98) - open/close 6418 (time 1787839680.74 total 278.87 last 26.29) - open/close 6641 (time 1787839690.74 total 288.87 last 22.29) - open/close 6860 (time 1787839701.66 total 299.79 last 20.06) - open/close 7083 (time 1787839712.59 total 310.72 last 20.41) - open/close 7322 (time 1787839722.60 total 320.73 last 23.87) - open/close 7581 (time 1787839732.61 total 330.74 last 25.88) - open/close 7789 (time 1787839742.86 total 340.99 last 20.28) - open/close 8006 (time 1787839753.24 total 351.37 last 20.91) - open/close 8207 (time 1787839763.25 total 361.38 last 20.07) - open/close 8454 (time 1787839773.26 total 371.39 last 24.68) - open/close 8677 (time 1787839783.27 total 381.40 last 22.28) - open/close 9011 (time 1787839793.91 total 392.04 last 31.40) - open/close 9245 (time 1787839803.94 total 402.07 last 23.33) - open/close 9493 (time 1787839813.94 total 412.07 last 24.79) - open/close 9756 (time 1787839824.60 total 422.73 last 24.67) - open/close 10000 (time 1787839834.21 total 432.34 last 25.39) - open/close 10171 (time 1787839844.89 total 443.02 last 16.01) - open/close 10365 (time 1787839854.90 total 453.04 last 19.37) - open/close 10642 (time 1787839865.24 total 463.37 last 26.80) - open/close 11027 (time 1787839875.24 total 473.37 last 38.50) - open/close 11296 (time 1787839887.63 total 485.76 last 21.71) - open/close 11476 (time 1787839898.07 total 496.20 last 17.24) - open/close 11702 (time 1787839908.31 total 506.44 last 22.08) - open/close 11881 (time 1787839918.32 total 516.45 last 17.88) - open/close 12117 (time 1787839928.91 total 527.04 last 22.28) - open/close 12323 (time 1787839939.63 total 537.76 last 19.22) - open/close 12560 (time 1787839950.67 total 548.80 last 21.46) - open/close 12727 (time 1787839960.67 total 558.80 last 16.70) - open/close 12967 (time 1787839970.68 total 568.81 last 23.98) - open/close 13164 (time 1787839980.69 total 578.82 last 19.69) - open/close 13310 (time 1787839990.69 total 588.82 last 14.60) - open/close 13462 (time 1787840000.69 total 598.82 last 15.19) - open/close 13641 (time 1787840011.35 total 609.48 last 16.80) - open/close 13884 (time 1787840022.16 total 620.29 last 22.47) - open/close 14127 (time 1787840033.10 total 631.24 last 22.21) - open/close 14409 (time 1787840043.66 total 641.79 last 26.71) - open/close 14702 (time 1787840053.67 total 651.80 last 29.27) - open/close 14990 (time 1787840064.46 total 662.59 last 26.69) - open/close 15218 (time 1787840074.47 total 672.60 last 22.79) - open/close 15468 (time 1787840084.47 total 682.60 last 24.98) - open/close 15679 (time 1787840094.68 total 692.81 last 20.68) - open/close 15900 (time 1787840105.10 total 703.23 last 21.20) - open/close 16104 (time 1787840115.47 total 713.60 last 19.67) - open/close 16350 (time 1787840125.47 total 723.60 last 24.60) - open/close 16533 (time 1787840135.47 total 733.60 last 18.30) - open/close 16781 (time 1787840146.23 total 744.36 last 23.05) - open/close 17045 (time 1787840157.07 total 755.20 last 24.36) - open/close 17297 (time 1787840167.26 total 765.39 last 24.72) - open/close 17523 (time 1787840177.27 total 775.40 last 22.57) - open/close 17811 (time 1787840187.29 total 785.42 last 28.77) - open/close 18069 (time 1787840197.40 total 795.53 last 25.52) - open/close 18291 (time 1787840207.86 total 805.99 last 21.23) - open/close 18415 (time 1787840218.66 total 816.80 last 11.47) - open/close 18547 (time 1787840229.33 total 827.46 last 12.37) - open/close 18817 (time 1787840239.82 total 837.95 last 25.73) - open/close 19069 (time 1787840250.12 total 848.25 last 24.47) - open/close 19341 (time 1787840260.63 total 858.76 last 25.90) - open/close 20000 (time 1787840269.69 total 867.82 last 72.70) - open/close 20752 (time 1787840279.69 total 877.82 last 75.20) - open/close 21541 (time 1787840289.70 total 887.84 last 78.79) - open/close 22273 (time 1787840299.72 total 897.85 last 73.08) - open/close 22962 (time 1787840309.73 total 907.86 last 68.84) - open/close 23609 (time 1787840319.73 total 917.86 last 64.67) - open/close 24390 (time 1787840329.74 total 927.87 last 78.05) - open/close 25108 (time 1787840339.74 total 937.87 last 71.79) - open/close 25797 (time 1787840349.77 total 947.90 last 68.68) - open/close 26545 (time 1787840359.78 total 957.91 last 74.76) - open/close 27266 (time 1787840369.78 total 967.91 last 72.08) - open/close 27984 (time 1787840379.79 total 977.92 last 71.76) - open/close 28702 (time 1787840389.79 total 987.92 last 71.77) - open/close 29322 (time 1787840399.80 total 997.93 last 61.93) - open/close 30000 (time 1787840409.03 total 1007.16 last 73.51) - open/close 30777 (time 1787840419.04 total 1017.17 last 77.58) - open/close 31547 (time 1787840429.06 total 1027.19 last 76.84) - open/close 32369 (time 1787840439.07 total 1037.20 last 82.16) - open/close 33122 (time 1787840449.08 total 1047.21 last 75.21) - open/close 33838 (time 1787840459.09 total 1057.22 last 71.51) - open/close 34586 (time 1787840469.10 total 1067.23 last 74.77) - open/close 35356 (time 1787840479.10 total 1077.23 last 76.94) - open/close 36179 (time 1787840489.11 total 1087.24 last 82.27) - open/close 36810 (time 1787840499.12 total 1097.25 last 63.01) - open/close 37417 (time 1787840509.13 total 1107.26 last 60.62) - open/close 38155 (time 1787840519.14 total 1117.27 last 73.77) - open/close 38933 (time 1787840529.14 total 1127.27 last 77.77) - open/close 39719 (time 1787840539.16 total 1137.29 last 78.49) - open/close 40000 (time 1787840542.68 total 1140.81 last 79.69) - open/close 40672 (time 1787840552.69 total 1150.82 last 67.16) - open/close 41320 (time 1787840562.70 total 1160.83 last 64.74) - open/close 42008 (time 1787840572.70 total 1170.83 last 68.78) - open/close 42717 (time 1787840582.71 total 1180.84 last 70.86) - open/close 43442 (time 1787840592.72 total 1190.85 last 72.39) - open/close 44162 (time 1787840602.73 total 1200.86 last 71.98) - open/close 44913 (time 1787840612.73 total 1210.86 last 75.06) - open/close 45699 (time 1787840622.73 total 1220.86 last 78.59) - open/close 46498 (time 1787840632.75 total 1230.88 last 79.75) - open/close 47261 (time 1787840642.75 total 1240.88 last 76.28) - open/close 48066 (time 1787840652.76 total 1250.89 last 80.44) - open/close 49118 (time 1787840662.77 total 1260.90 last 105.16) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1267.35 seconds: 39.16 ops/second - unlinked 0 (time 1787840672 ; total 0 ; last 0) - unlinked 10000 (time 1787840800 ; total 128 ; last 128) - unlinked 20000 (time 1787840921 ; total 249 ; last 121) - unlinked 30000 (time 1787841047 ; total 375 ; last 126) - unlinked 40000 (time 1787841179 ; total 507 ; last 132) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 631 seconds: 78.654518 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg619-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg619-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg619-server stop mds service on oleg619-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg619-server stop mds service on oleg619-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg619-server unloading modules via unload_modules_local on: 'oleg619-server' oleg619-server: oleg619-server.virtnet: executing unload_modules_local oleg619-server: modules unloaded. pdsh@oleg619-client: oleg619-client: ssh exited with exit code 2 pdsh@oleg619-client: oleg619-server: ssh exited with exit code 2 pdsh@oleg619-client: oleg619-client: ssh exited with exit code 2 pdsh@oleg619-client: oleg619-server: ssh exited with exit code 2