== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:52:02 (1787377922) start mds service on oleg247-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg247-server' oleg247-server: oleg247-server.virtnet: executing load_modules_local oleg247-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg247-server: Loading modules from /home/green/git/lustre-release/lustre oleg247-server: detected 4 online CPUs by sysfs oleg247-server: MODOPTS_LIBCFS= oleg247-server: Force libcfs to create 2 CPU partitions oleg247-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg247-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg247-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg247-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg247-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-OST0000 oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg247-client.virtnet: -o user_xattr,flock 192.168.202.147@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 272 (time 1787378037.15 total 10.01 last 27.18) - open/close 495 (time 1787378047.86 total 20.72 last 20.81) - open/close 744 (time 1787378057.88 total 30.74 last 24.86) - open/close 950 (time 1787378068.08 total 40.94 last 20.19) - open/close 1206 (time 1787378078.31 total 51.17 last 25.04) - open/close 1506 (time 1787378088.31 total 61.17 last 29.98) - open/close 1770 (time 1787378098.34 total 71.20 last 26.33) - open/close 1966 (time 1787378108.49 total 81.35 last 19.31) - open/close 2179 (time 1787378119.19 total 92.05 last 19.90) - open/close 2382 (time 1787378129.21 total 102.07 last 20.26) - open/close 2638 (time 1787378139.58 total 112.44 last 24.69) - open/close 2865 (time 1787378149.59 total 122.45 last 22.68) - open/close 3097 (time 1787378159.97 total 132.83 last 22.35) - open/close 3345 (time 1787378170.36 total 143.22 last 23.85) - open/close 3591 (time 1787378180.79 total 153.65 last 23.60) - open/close 3903 (time 1787378191.42 total 164.28 last 29.36) - open/close 4169 (time 1787378202.27 total 175.13 last 24.50) - open/close 4423 (time 1787378212.32 total 185.18 last 25.28) - open/close 4731 (time 1787378222.47 total 195.33 last 30.34) - open/close 4931 (time 1787378233.34 total 206.20 last 18.39) - open/close 5217 (time 1787378244.35 total 217.21 last 25.98) - open/close 5495 (time 1787378254.36 total 227.22 last 27.77) - open/close 5649 (time 1787378264.37 total 237.23 last 15.39) - open/close 5943 (time 1787378274.49 total 247.35 last 29.03) - open/close 6207 (time 1787378284.50 total 257.36 last 26.39) - open/close 6464 (time 1787378294.50 total 267.36 last 25.69) - open/close 6669 (time 1787378304.51 total 277.37 last 20.48) - open/close 6925 (time 1787378315.21 total 288.07 last 23.93) - open/close 7013 (time 1787378326.20 total 299.06 last 8.00) - open/close 7137 (time 1787378337.60 total 310.46 last 10.88) - open/close 7365 (time 1787378348.18 total 321.04 last 21.56) - open/close 7593 (time 1787378358.20 total 331.06 last 22.76) - open/close 7847 (time 1787378368.91 total 341.77 last 23.71) - open/close 8147 (time 1787378378.92 total 351.78 last 29.98) - open/close 8364 (time 1787378388.93 total 361.79 last 21.68) - open/close 8402 (time 1787378399.93 total 372.79 last 3.45) - open/close 8552 (time 1787378410.07 total 382.93 last 14.81) - open/close 8678 (time 1787378420.22 total 393.08 last 12.41) - open/close 8819 (time 1787378432.63 total 405.49 last 11.36) - open/close 8960 (time 1787378442.67 total 415.53 last 14.04) - open/close 9143 (time 1787378453.74 total 426.60 last 16.53) - open/close 9319 (time 1787378464.59 total 437.45 last 16.22) - open/close 9513 (time 1787378474.75 total 447.61 last 19.10) - open/close 9619 (time 1787378485.56 total 458.42 last 9.81) - open/close 9726 (time 1787378495.57 total 468.43 last 10.68) - open/close 9949 (time 1787378506.30 total 479.16 last 20.79) - open/close 10238 (time 1787378516.30 total 489.16 last 28.89) - open/close 10455 (time 1787378526.30 total 499.16 last 21.70) - open/close 10674 (time 1787378536.32 total 509.18 last 21.86) - open/close 10948 (time 1787378546.33 total 519.19 last 27.39) - open/close 11126 (time 1787378556.33 total 529.19 last 17.80) - open/close 11339 (time 1787378566.34 total 539.20 last 21.26) - open/close 11559 (time 1787378576.72 total 549.58 last 21.20) - open/close 11802 (time 1787378587.76 total 560.62 last 22.00) - open/close 12101 (time 1787378597.79 total 570.64 last 29.83) - open/close 12353 (time 1787378607.81 total 580.67 last 25.15) - open/close 12578 (time 1787378617.81 total 590.67 last 22.49) - open/close 12807 (time 1787378628.27 total 601.13 last 21.89) - open/close 13015 (time 1787378638.29 total 611.14 last 20.77) - open/close 13260 (time 1787378649.29 total 622.15 last 22.27) - open/close 13450 (time 1787378659.31 total 632.17 last 18.95) - open/close 13569 (time 1787378669.82 total 642.68 last 11.32) - open/close 13825 (time 1787378679.87 total 652.73 last 25.47) - open/close 13995 (time 1787378690.56 total 663.42 last 15.90) - open/close 14222 (time 1787378700.81 total 673.67 last 22.16) - open/close 14433 (time 1787378710.96 total 683.82 last 20.78) - open/close 14684 (time 1787378720.96 total 693.82 last 25.10) - open/close 14911 (time 1787378731.38 total 704.24 last 21.79) - open/close 15174 (time 1787378742.26 total 715.12 last 24.18) - open/close 15469 (time 1787378753.03 total 725.88 last 27.40) - open/close 15722 (time 1787378763.03 total 735.89 last 25.29) - open/close 16088 (time 1787378773.05 total 745.91 last 36.52) - open/close 16330 (time 1787378783.48 total 756.34 last 23.20) - open/close 16478 (time 1787378793.67 total 766.53 last 14.52) - open/close 16552 (time 1787378803.67 total 776.53 last 7.40) - open/close 16632 (time 1787378814.71 total 787.57 last 7.25) - open/close 16969 (time 1787378824.73 total 797.59 last 33.65) - open/close 17221 (time 1787378834.74 total 807.60 last 25.16) - open/close 17515 (time 1787378847.61 total 820.47 last 22.84) - open/close 17735 (time 1787378857.62 total 830.48 last 21.99) - open/close 17915 (time 1787378868.22 total 841.08 last 16.98) - open/close 18216 (time 1787378879.04 total 851.90 last 27.80) - open/close 18478 (time 1787378889.93 total 862.79 last 24.06) - open/close 18735 (time 1787378900.54 total 873.40 last 24.22) - open/close 18999 (time 1787378911.36 total 884.22 last 24.41) - open/close 19174 (time 1787378926.21 total 899.07 last 11.78) - open/close 19418 (time 1787378938.74 total 911.60 last 19.48) - open/close 19641 (time 1787378949.24 total 922.10 last 21.25) - open/close 19886 (time 1787378959.25 total 932.11 last 24.47) - open/close 20000 (time 1787378964.62 total 937.48 last 21.22) - open/close 20205 (time 1787378974.63 total 947.49 last 20.48) - open/close 20326 (time 1787378985.11 total 957.97 last 11.55) - open/close 20453 (time 1787378995.13 total 967.99 last 12.68) - open/close 20619 (time 1787379005.63 total 978.49 last 15.81) - open/close 20785 (time 1787379015.63 total 988.49 last 16.60) - open/close 21466 (time 1787379025.63 total 998.49 last 68.07) - open/close 22434 (time 1787379035.64 total 1008.50 last 96.71) - open/close 23369 (time 1787379045.65 total 1018.51 last 93.46) - open/close 24294 (time 1787379055.65 total 1028.51 last 92.48) - open/close 25142 (time 1787379065.65 total 1038.51 last 84.79) - open/close 26112 (time 1787379075.66 total 1048.52 last 96.94) - open/close 27147 (time 1787379085.66 total 1058.52 last 103.48) - open/close 28101 (time 1787379095.67 total 1068.53 last 95.33) - open/close 28843 (time 1787379105.67 total 1078.53 last 74.16) - open/close 29633 (time 1787379115.68 total 1088.54 last 78.95) - open/close 30000 (time 1787379119.66 total 1092.52 last 92.05) - open/close 30734 (time 1787379129.67 total 1102.53 last 73.39) - open/close 31449 (time 1787379139.67 total 1112.53 last 71.44) - open/close 32315 (time 1787379149.68 total 1122.54 last 86.58) - open/close 33191 (time 1787379159.69 total 1132.55 last 87.51) - open/close 34048 (time 1787379169.70 total 1142.56 last 85.58) - open/close 34963 (time 1787379179.71 total 1152.57 last 91.45) - open/close 35768 (time 1787379189.71 total 1162.57 last 80.43) - open/close 36546 (time 1787379199.72 total 1172.58 last 77.75) - open/close 37386 (time 1787379209.72 total 1182.58 last 83.99) - open/close 38234 (time 1787379219.72 total 1192.58 last 84.78) - open/close 39043 (time 1787379229.73 total 1202.59 last 80.88) - open/close 39936 (time 1787379239.73 total 1212.59 last 89.30) - open/close 40504 (time 1787379249.73 total 1222.59 last 56.77) - open/close 41457 (time 1787379259.74 total 1232.60 last 95.25) - open/close 42394 (time 1787379269.74 total 1242.60 last 93.65) - open/close 43344 (time 1787379279.75 total 1252.61 last 94.96) - open/close 44224 (time 1787379289.75 total 1262.61 last 88.00) - open/close 45169 (time 1787379299.75 total 1272.61 last 94.44) - open/close 46181 (time 1787379309.76 total 1282.62 last 101.15) - open/close 47117 (time 1787379319.76 total 1292.62 last 93.54) - open/close 48025 (time 1787379329.77 total 1302.63 last 90.79) - open/close 48952 (time 1787379339.78 total 1312.64 last 92.57) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1320.52 seconds: 37.58 ops/second - unlinked 0 (time 1787379352 ; total 0 ; last 0) - unlinked 10000 (time 1787379463 ; total 111 ; last 111) - unlinked 20000 (time 1787379581 ; total 229 ; last 118) - unlinked 30000 (time 1787379645 ; total 293 ; last 64) - unlinked 40000 (time 1787379713 ; total 361 ; last 68) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 424 seconds: 117.054245 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg247-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg247-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg247-server stop mds service on oleg247-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg247-server stop mds service on oleg247-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg247-server unloading modules via unload_modules_local on: 'oleg247-server' oleg247-server: oleg247-server.virtnet: executing unload_modules_local oleg247-server: modules unloaded. pdsh@oleg247-client: oleg247-client: ssh exited with exit code 2 pdsh@oleg247-client: oleg247-server: ssh exited with exit code 2 pdsh@oleg247-client: oleg247-server: ssh exited with exit code 2 pdsh@oleg247-client: oleg247-client: ssh exited with exit code 2