== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 17:31:22 (1778621482) start mds service on oleg121-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg121-server' oleg121-server: oleg121-server.virtnet: executing load_modules_local oleg121-server: Loading modules from /home/green/git/lustre-release/lustre oleg121-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg121-server: detected 4 online CPUs by sysfs oleg121-server: MODOPTS_LIBCFS= oleg121-server: Force libcfs to create 2 CPU partitions oleg121-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg121-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg121-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg121-server: oleg121-server.virtnet: executing set_default_debug -1 all pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg121-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg121-server: oleg121-server.virtnet: executing set_default_debug -1 all pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg121-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg121-server: oleg121-server.virtnet: executing set_default_debug -1 all pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 Started lustre-OST0000 oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg121-client.virtnet: -o user_xattr,flock 192.168.201.121@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 146 (time 1778621588.68 total 11.65 last 12.53) - open/close 325 (time 1778621599.40 total 22.37 last 16.70) - open/close 444 (time 1778621609.41 total 32.38 last 11.89) - open/close 578 (time 1778621620.03 total 43.00 last 12.62) - open/close 804 (time 1778621630.33 total 53.30 last 21.94) - open/close 919 (time 1778621640.98 total 63.95 last 10.80) - open/close 1096 (time 1778621650.99 total 73.96 last 17.69) - open/close 1222 (time 1778621661.09 total 84.06 last 12.47) - open/close 1395 (time 1778621671.47 total 94.44 last 16.67) - open/close 1523 (time 1778621682.32 total 105.29 last 11.80) - open/close 1647 (time 1778621695.39 total 118.36 last 9.48) - open/close 1798 (time 1778621706.05 total 129.02 last 14.16) - open/close 2032 (time 1778621716.87 total 139.84 last 21.63) - open/close 2211 (time 1778621726.88 total 149.85 last 17.89) - open/close 2296 (time 1778621739.12 total 162.09 last 6.94) - open/close 2480 (time 1778621749.16 total 172.13 last 18.32) - open/close 2682 (time 1778621759.68 total 182.65 last 19.20) - open/close 2850 (time 1778621769.69 total 192.66 last 16.78) - open/close 2926 (time 1778621779.93 total 202.90 last 7.43) - open/close 3095 (time 1778621790.47 total 213.44 last 16.03) - open/close 3269 (time 1778621800.51 total 223.48 last 17.33) - open/close 3453 (time 1778621810.51 total 233.48 last 18.40) - open/close 3619 (time 1778621820.51 total 243.48 last 16.59) - open/close 3775 (time 1778621832.86 total 255.83 last 12.63) - open/close 4027 (time 1778621842.88 total 265.85 last 25.15) - open/close 4281 (time 1778621852.89 total 275.86 last 25.38) - open/close 4468 (time 1778621863.69 total 286.66 last 17.32) - open/close 4666 (time 1778621874.13 total 297.10 last 18.96) - open/close 4794 (time 1778621884.36 total 307.33 last 12.52) - open/close 4971 (time 1778621894.98 total 317.95 last 16.66) - open/close 5204 (time 1778621906.51 total 329.48 last 20.21) - open/close 5309 (time 1778621916.55 total 339.52 last 10.46) - open/close 5477 (time 1778621926.93 total 349.90 last 16.18) - open/close 5634 (time 1778621937.25 total 360.22 last 15.22) - open/close 5751 (time 1778621947.72 total 370.69 last 11.17) - open/close 5872 (time 1778621957.73 total 380.70 last 12.09) - open/close 5980 (time 1778621967.94 total 390.91 last 10.58) - open/close 6084 (time 1778621977.98 total 400.95 last 10.36) - open/close 6185 (time 1778621988.40 total 411.37 last 9.70) - open/close 6422 (time 1778621999.11 total 422.08 last 22.11) - open/close 6594 (time 1778622009.14 total 432.11 last 17.16) - open/close 6833 (time 1778622019.16 total 442.13 last 23.86) - open/close 6919 (time 1778622029.54 total 452.51 last 8.28) - open/close 7046 (time 1778622039.55 total 462.52 last 12.68) - open/close 7134 (time 1778622049.55 total 472.52 last 8.80) - open/close 7286 (time 1778622060.48 total 483.45 last 13.91) - open/close 7348 (time 1778622071.28 total 494.25 last 5.74) - open/close 7456 (time 1778622082.13 total 505.10 last 9.95) - open/close 7629 (time 1778622092.33 total 515.30 last 16.97) - open/close 7713 (time 1778622102.85 total 525.82 last 7.98) - open/close 7776 (time 1778622112.88 total 535.85 last 6.29) - open/close 7935 (time 1778622125.45 total 548.42 last 12.65) - open/close 8147 (time 1778622135.83 total 558.80 last 20.43) - open/close 8265 (time 1778622145.84 total 568.81 last 11.79) - open/close 8498 (time 1778622156.74 total 579.71 last 21.37) - open/close 8706 (time 1778622166.76 total 589.73 last 20.77) - open/close 8811 (time 1778622176.77 total 599.74 last 10.48) - open/close 8878 (time 1778622187.84 total 610.81 last 6.05) - open/close 8973 (time 1778622200.45 total 623.42 last 7.53) - open/close 9067 (time 1778622210.45 total 633.42 last 9.40) - open/close 9145 (time 1778622221.00 total 643.97 last 7.40) - open/close 9245 (time 1778622231.78 total 654.75 last 9.27) - open/close 9328 (time 1778622242.59 total 665.56 last 7.68) - open/close 9391 (time 1778622252.68 total 675.65 last 6.24) - open/close 9606 (time 1778622262.69 total 685.67 last 21.47) - open/close 9841 (time 1778622273.68 total 696.65 last 21.38) - open/close 9883 (time 1778622284.17 total 707.14 last 4.01) - open/close 10000 (time 1778622290.08 total 713.05 last 19.79) - open/close 10125 (time 1778622300.10 total 723.07 last 12.47) - open/close 10344 (time 1778622310.11 total 733.08 last 21.88) - open/close 10541 (time 1778622320.52 total 743.49 last 18.91) - open/close 10674 (time 1778622330.97 total 753.94 last 12.73) - open/close 10786 (time 1778622341.32 total 764.30 last 10.82) - open/close 10893 (time 1778622351.33 total 774.30 last 10.69) - open/close 10956 (time 1778622361.35 total 784.32 last 6.29) - open/close 11160 (time 1778622372.18 total 795.15 last 18.84) - open/close 11294 (time 1778622382.87 total 805.84 last 12.53) - open/close 11422 (time 1778622393.75 total 816.72 last 11.76) - open/close 11528 (time 1778622404.80 total 827.77 last 9.59) - open/close 11625 (time 1778622415.30 total 838.27 last 9.24) - open/close 11815 (time 1778622425.61 total 848.58 last 18.43) - open/close 11908 (time 1778622436.50 total 859.47 last 8.54) - open/close 12026 (time 1778622447.07 total 870.04 last 11.16) - open/close 12070 (time 1778622457.76 total 880.73 last 4.12) - open/close 12184 (time 1778622467.77 total 890.74 last 11.39) - open/close 12291 (time 1778622478.54 total 901.51 last 9.93) - open/close 12381 (time 1778622488.57 total 911.54 last 8.98) - open/close 12462 (time 1778622499.34 total 922.31 last 7.52) - open/close 12563 (time 1778622510.28 total 933.25 last 9.24) - open/close 12631 (time 1778622521.39 total 944.36 last 6.12) - open/close 12806 (time 1778622534.09 total 957.06 last 13.77) - open/close 12897 (time 1778622544.92 total 967.89 last 8.40) - open/close 13029 (time 1778622555.73 total 978.70 last 12.22) - open/close 13186 (time 1778622565.87 total 988.84 last 15.48) - open/close 13301 (time 1778622575.88 total 998.85 last 11.49) - open/close 13434 (time 1778622586.70 total 1009.67 last 12.29) - open/close 13627 (time 1778622596.72 total 1019.69 last 19.27) - open/close 13726 (time 1778622607.13 total 1030.10 last 9.50) - open/close 13826 (time 1778622617.14 total 1040.12 last 9.99) - open/close 13970 (time 1778622627.15 total 1050.12 last 14.39) - open/close 14078 (time 1778622638.22 total 1061.19 last 9.75) - open/close 14161 (time 1778622648.40 total 1071.37 last 8.16) - open/close 14251 (time 1778622659.04 total 1082.01 last 8.45) - open/close 14359 (time 1778622669.22 total 1092.19 last 10.61) - open/close 14469 (time 1778622679.22 total 1102.19 last 11.00) - open/close 14556 (time 1778622689.90 total 1112.87 last 8.14) - open/close 14672 (time 1778622700.68 total 1123.65 last 10.76) - open/close 14879 (time 1778622710.70 total 1133.67 last 20.68) - open/close 14973 (time 1778622721.54 total 1144.51 last 8.67) - open/close 15077 (time 1778622732.46 total 1155.43 last 9.52) - open/close 15184 (time 1778622743.18 total 1166.15 last 9.98) - open/close 15497 (time 1778622753.19 total 1176.16 last 31.26) - open/close 16101 (time 1778622763.20 total 1186.17 last 60.37) - open/close 16696 (time 1778622773.21 total 1196.18 last 59.44) - open/close 17223 (time 1778622783.23 total 1206.20 last 52.60) - open/close 17841 (time 1778622793.26 total 1216.23 last 61.57) - open/close 18582 (time 1778622803.26 total 1226.23 last 74.10) - open/close 19265 (time 1778622813.27 total 1236.24 last 68.23) - open/close 20000 (time 1778622821.92 total 1244.90 last 84.95) - open/close 20875 (time 1778622831.93 total 1254.90 last 87.45) - open/close 21578 (time 1778622841.93 total 1264.90 last 70.28) - open/close 22435 (time 1778622851.95 total 1274.92 last 85.58) - open/close 23090 (time 1778622861.96 total 1284.93 last 65.45) - open/close 23783 (time 1778622871.96 total 1294.93 last 69.30) - open/close 24417 (time 1778622881.99 total 1304.96 last 63.18) - open/close 25031 (time 1778622892.00 total 1314.97 last 61.36) - open/close 25702 (time 1778622902.00 total 1324.97 last 67.07) - open/close 26456 (time 1778622912.01 total 1334.98 last 75.36) - open/close 27193 (time 1778622922.01 total 1344.98 last 73.67) - open/close 27893 (time 1778622932.02 total 1354.99 last 69.91) - open/close 28582 (time 1778622942.03 total 1365.00 last 68.84) - open/close 29198 (time 1778622952.03 total 1375.00 last 61.60) - open/close 29928 (time 1778622962.05 total 1385.02 last 72.85) - open/close 30626 (time 1778622972.06 total 1395.03 last 69.75) - open/close 31287 (time 1778622982.08 total 1405.05 last 65.99) - open/close 32035 (time 1778622992.08 total 1415.05 last 74.76) - open/close 32697 (time 1778623002.08 total 1425.05 last 66.18) - open/close 33345 (time 1778623012.09 total 1435.06 last 64.79) - open/close 33978 (time 1778623022.10 total 1445.07 last 63.20) - open/close 34576 (time 1778623032.11 total 1455.08 last 59.73) - open/close 35332 (time 1778623042.12 total 1465.09 last 75.56) - open/close 36153 (time 1778623052.13 total 1475.10 last 82.04) - open/close 36863 (time 1778623062.14 total 1485.11 last 70.90) - open/close 37385 (time 1778623072.15 total 1495.12 last 52.13) - open/close 38056 (time 1778623082.18 total 1505.15 last 66.92) - open/close 38757 (time 1778623092.20 total 1515.17 last 69.99) - open/close 39323 (time 1778623102.20 total 1525.17 last 56.56) - open/close 39850 (time 1778623112.21 total 1535.18 last 52.67) - open/close 40000 (time 1778623115.67 total 1538.64 last 43.30) - open/close 40533 (time 1778623125.68 total 1548.65 last 53.27) - open/close 41214 (time 1778623135.70 total 1558.67 last 67.95) - open/close 41841 (time 1778623145.71 total 1568.68 last 62.62) - open/close 42540 (time 1778623155.71 total 1578.68 last 69.89) - open/close 43212 (time 1778623165.73 total 1588.70 last 67.11) - open/close 43795 (time 1778623175.74 total 1598.71 last 58.20) - open/close 44429 (time 1778623185.75 total 1608.72 last 63.34) - open/close 45042 (time 1778623195.76 total 1618.73 last 61.23) - open/close 45553 (time 1778623205.78 total 1628.75 last 51.03) - open/close 46204 (time 1778623215.78 total 1638.75 last 65.08) - open/close 46794 (time 1778623225.79 total 1648.76 last 58.93) - open/close 47344 (time 1778623235.79 total 1658.76 last 54.99) - open/close 47984 (time 1778623245.80 total 1668.77 last 63.96) - open/close 48536 (time 1778623255.81 total 1678.78 last 55.14) - open/close 49000 (time 1778623265.84 total 1688.81 last 46.28) - open/close 49472 (time 1778623275.85 total 1698.82 last 47.14) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1702.29 seconds: 29.16 ops/second - unlinked 0 (time 1778623286 ; total 0 ; last 0) - unlinked 10000 (time 1778623449 ; total 163 ; last 163) - unlinked 20000 (time 1778623620 ; total 334 ; last 171) - unlinked 30000 (time 1778623780 ; total 494 ; last 160) - unlinked 40000 (time 1778623946 ; total 660 ; last 166) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 823 seconds: 60.306198 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg121-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg121-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg121-server stop mds service on oleg121-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg121-server stop mds service on oleg121-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg121-server unloading modules via unload_modules_local on: 'oleg121-server' oleg121-server: oleg121-server.virtnet: executing unload_modules_local oleg121-server: modules unloaded. pdsh@oleg121-client: oleg121-client: ssh exited with exit code 2 pdsh@oleg121-client: oleg121-server: ssh exited with exit code 2 pdsh@oleg121-client: oleg121-client: ssh exited with exit code 2 pdsh@oleg121-client: oleg121-server: ssh exited with exit code 2