== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:48:29 (1779979709) start mds service on oleg208-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg208-server' oleg208-server: oleg208-server.virtnet: executing load_modules_local oleg208-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg208-server: Loading modules from /home/green/git/lustre-release/lustre oleg208-server: detected 4 online CPUs by sysfs oleg208-server: MODOPTS_LIBCFS= oleg208-server: Force libcfs to create 2 CPU partitions oleg208-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg208-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg208-server: oleg208-server.virtnet: executing set_default_debug -1 all pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg208-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg208-server: oleg208-server.virtnet: executing set_default_debug -1 all pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg208-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg208-server: oleg208-server.virtnet: executing set_default_debug -1 all pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Started lustre-OST0000 oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg208-client.virtnet: -o user_xattr,flock 192.168.202.108@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 304 (time 1779979804.26 total 10.48 last 29.02) - open/close 582 (time 1779979814.26 total 20.48 last 27.79) - open/close 850 (time 1779979824.86 total 31.08 last 25.29) - open/close 961 (time 1779979834.86 total 41.08 last 11.10) - open/close 1099 (time 1779979845.79 total 52.01 last 12.62) - open/close 1272 (time 1779979856.23 total 62.45 last 16.58) - open/close 1524 (time 1779979866.23 total 72.45 last 25.19) - open/close 1722 (time 1779979877.12 total 83.34 last 18.19) - open/close 1865 (time 1779979887.13 total 93.36 last 14.27) - open/close 2008 (time 1779979897.64 total 103.86 last 13.61) - open/close 2130 (time 1779979908.57 total 114.79 last 11.16) - open/close 2377 (time 1779979919.22 total 125.44 last 23.20) - open/close 2704 (time 1779979929.52 total 135.74 last 31.74) - open/close 2864 (time 1779979939.90 total 146.12 last 15.41) - open/close 3030 (time 1779979949.91 total 156.13 last 16.58) - open/close 3185 (time 1779979959.94 total 166.16 last 15.45) - open/close 3458 (time 1779979970.23 total 176.45 last 26.54) - open/close 3630 (time 1779979980.77 total 186.99 last 16.33) - open/close 3742 (time 1779979990.77 total 197.00 last 11.19) - open/close 3834 (time 1779980000.79 total 207.01 last 9.18) - open/close 3990 (time 1779980011.30 total 217.52 last 14.85) - open/close 4159 (time 1779980023.87 total 230.10 last 13.44) - open/close 4437 (time 1779980034.92 total 241.14 last 25.17) - open/close 4648 (time 1779980044.93 total 251.15 last 21.08) - open/close 4830 (time 1779980055.34 total 261.56 last 17.47) - open/close 4983 (time 1779980065.37 total 271.59 last 15.27) - open/close 5092 (time 1779980078.31 total 284.53 last 8.42) - open/close 5240 (time 1779980088.61 total 294.83 last 14.37) - open/close 5441 (time 1779980098.85 total 305.07 last 19.64) - open/close 5514 (time 1779980108.90 total 315.13 last 7.26) - open/close 5785 (time 1779980118.91 total 325.13 last 27.09) - open/close 5976 (time 1779980128.94 total 335.16 last 19.04) - open/close 6234 (time 1779980139.61 total 345.83 last 24.18) - open/close 6431 (time 1779980150.30 total 356.52 last 18.42) - open/close 6597 (time 1779980160.32 total 366.54 last 16.56) - open/close 6880 (time 1779980170.57 total 376.79 last 27.63) - open/close 7140 (time 1779980180.58 total 386.80 last 25.95) - open/close 7322 (time 1779980191.27 total 397.49 last 17.03) - open/close 7563 (time 1779980202.33 total 408.55 last 21.79) - open/close 7639 (time 1779980212.34 total 418.56 last 7.59) - open/close 7945 (time 1779980222.34 total 428.56 last 30.58) - open/close 8155 (time 1779980232.74 total 438.96 last 20.21) - open/close 8392 (time 1779980245.02 total 451.24 last 19.30) - open/close 8465 (time 1779980255.39 total 461.62 last 7.03) - open/close 8591 (time 1779980265.40 total 471.62 last 12.59) - open/close 8792 (time 1779980275.40 total 481.63 last 20.09) - open/close 9076 (time 1779980285.85 total 492.07 last 27.18) - open/close 9370 (time 1779980296.09 total 502.31 last 28.71) - open/close 9637 (time 1779980306.45 total 512.67 last 25.77) - open/close 9852 (time 1779980317.15 total 523.37 last 20.10) - open/close 10000 (time 1779980323.54 total 529.76 last 23.16) - open/close 10132 (time 1779980333.56 total 539.78 last 13.18) - open/close 10288 (time 1779980343.78 total 550.00 last 15.27) - open/close 10457 (time 1779980354.83 total 561.06 last 15.28) - open/close 10611 (time 1779980367.08 total 573.30 last 12.58) - open/close 10852 (time 1779980377.08 total 583.30 last 24.10) - open/close 10995 (time 1779980387.13 total 593.35 last 14.23) - open/close 11250 (time 1779980397.16 total 603.38 last 25.42) - open/close 11485 (time 1779980407.46 total 613.68 last 22.82) - open/close 11779 (time 1779980417.46 total 623.68 last 29.40) - open/close 12079 (time 1779980427.48 total 633.70 last 29.94) - open/close 12336 (time 1779980438.43 total 644.65 last 23.46) - open/close 12581 (time 1779980449.19 total 655.41 last 22.78) - open/close 12844 (time 1779980459.74 total 665.96 last 24.93) - open/close 13074 (time 1779980469.86 total 676.08 last 22.73) - open/close 13310 (time 1779980480.17 total 686.39 last 22.88) - open/close 13513 (time 1779980490.40 total 696.62 last 19.86) - open/close 13778 (time 1779980500.40 total 706.62 last 26.49) - open/close 14072 (time 1779980510.43 total 716.65 last 29.32) - open/close 14315 (time 1779980520.95 total 727.17 last 23.10) - open/close 14600 (time 1779980530.97 total 737.19 last 28.43) - open/close 14862 (time 1779980540.97 total 747.19 last 26.20) - open/close 15086 (time 1779980551.07 total 757.29 last 22.18) - open/close 15307 (time 1779980561.77 total 767.99 last 20.65) - open/close 15566 (time 1779980572.58 total 778.80 last 23.96) - open/close 15804 (time 1779980582.82 total 789.04 last 23.25) - open/close 16017 (time 1779980592.83 total 799.05 last 21.27) - open/close 16237 (time 1779980602.84 total 809.07 last 21.97) - open/close 16404 (time 1779980614.14 total 820.36 last 14.78) - open/close 16599 (time 1779980626.16 total 832.38 last 16.23) - open/close 16768 (time 1779980636.58 total 842.81 last 16.21) - open/close 17023 (time 1779980646.94 total 853.16 last 24.63) - open/close 17192 (time 1779980657.25 total 863.47 last 16.39) - open/close 17448 (time 1779980667.26 total 873.48 last 25.56) - open/close 17760 (time 1779980677.27 total 883.49 last 31.19) - open/close 17983 (time 1779980688.03 total 894.25 last 20.72) - open/close 18204 (time 1779980698.12 total 904.34 last 21.90) - open/close 18426 (time 1779980708.83 total 915.05 last 20.73) - open/close 18662 (time 1779980719.13 total 925.35 last 22.91) - open/close 18867 (time 1779980729.15 total 935.37 last 20.46) - open/close 19140 (time 1779980739.16 total 945.38 last 27.28) - open/close 19389 (time 1779980749.46 total 955.69 last 24.16) - open/close 19605 (time 1779980763.12 total 969.35 last 15.81) - open/close 19861 (time 1779980773.13 total 979.35 last 25.59) - open/close 20000 (time 1779980778.28 total 984.50 last 26.96) - open/close 20926 (time 1779980788.31 total 994.53 last 92.35) - open/close 21727 (time 1779980798.33 total 1004.56 last 79.90) - open/close 22667 (time 1779980808.34 total 1014.56 last 94.00) - open/close 23639 (time 1779980818.35 total 1024.57 last 97.04) - open/close 24674 (time 1779980828.36 total 1034.58 last 103.38) - open/close 25765 (time 1779980838.36 total 1044.59 last 109.08) - open/close 26601 (time 1779980848.37 total 1054.59 last 83.55) - open/close 27741 (time 1779980858.38 total 1064.60 last 113.88) - open/close 28803 (time 1779980868.38 total 1074.61 last 106.17) - open/close 30000 (time 1779980878.01 total 1084.23 last 124.33) - open/close 31002 (time 1779980888.02 total 1094.24 last 100.13) - open/close 32037 (time 1779980898.02 total 1104.24 last 103.49) - open/close 32862 (time 1779980908.03 total 1114.25 last 82.45) - open/close 33804 (time 1779980918.03 total 1124.25 last 94.20) - open/close 34681 (time 1779980928.03 total 1134.25 last 87.65) - open/close 35862 (time 1779980938.04 total 1144.26 last 118.07) - open/close 36839 (time 1779980948.04 total 1154.26 last 97.67) - open/close 37805 (time 1779980958.05 total 1164.27 last 96.52) - open/close 38821 (time 1779980968.05 total 1174.28 last 101.53) - open/close 39948 (time 1779980978.05 total 1184.28 last 112.70) - open/close 40876 (time 1779980988.06 total 1194.28 last 92.71) - open/close 41894 (time 1779980998.06 total 1204.29 last 101.79) - open/close 42868 (time 1779981008.06 total 1214.29 last 97.39) - open/close 43790 (time 1779981018.07 total 1224.29 last 92.18) - open/close 44738 (time 1779981028.08 total 1234.30 last 94.71) - open/close 45493 (time 1779981038.08 total 1244.30 last 75.49) - open/close 46243 (time 1779981048.08 total 1254.30 last 74.97) - open/close 47169 (time 1779981058.11 total 1264.33 last 92.36) - open/close 48118 (time 1779981068.11 total 1274.33 last 94.88) - open/close 49039 (time 1779981078.11 total 1284.33 last 92.07) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1291.61 seconds: 38.43 ops/second - unlinked 0 (time 1779981089 ; total 0 ; last 0) - unlinked 10000 (time 1779981203 ; total 114 ; last 114) - unlinked 20000 (time 1779981298 ; total 209 ; last 95) - unlinked 30000 (time 1779981364 ; total 275 ; last 66) - unlinked 40000 (time 1779981422 ; total 333 ; last 58) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 393 seconds: 126.290077 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg208-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg208-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg208-server stop mds service on oleg208-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg208-server stop mds service on oleg208-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg208-server unloading modules via unload_modules_local on: 'oleg208-server' oleg208-server: oleg208-server.virtnet: executing unload_modules_local oleg208-server: modules unloaded. pdsh@oleg208-client: oleg208-client: ssh exited with exit code 2 pdsh@oleg208-client: oleg208-server: ssh exited with exit code 2 pdsh@oleg208-client: oleg208-client: ssh exited with exit code 2 pdsh@oleg208-client: oleg208-server: ssh exited with exit code 2