== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 04:24:28 (1784190268) start mds service on oleg460-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg460-server' oleg460-server: oleg460-server.virtnet: executing load_modules_local oleg460-server: Loading modules from /home/green/git/lustre-release/lustre oleg460-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg460-server: detected 4 online CPUs by sysfs oleg460-server: MODOPTS_LIBCFS= oleg460-server: Force libcfs to create 2 CPU partitions oleg460-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg460-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg460-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg460-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg460-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-OST0000 oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg460-client.virtnet: -o user_xattr,flock 192.168.204.160@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 182 (time 1784190389.71 total 12.47 last 14.60) - open/close 320 (time 1784190400.02 total 22.78 last 13.38) - open/close 510 (time 1784190410.83 total 33.59 last 17.57) - open/close 723 (time 1784190422.93 total 45.68 last 17.62) - open/close 926 (time 1784190432.95 total 55.71 last 20.25) - open/close 1123 (time 1784190445.27 total 68.03 last 15.99) - open/close 1350 (time 1784190455.28 total 78.03 last 22.69) - open/close 1479 (time 1784190465.56 total 88.32 last 12.54) - open/close 1553 (time 1784190477.72 total 100.48 last 6.09) - open/close 1731 (time 1784190487.73 total 110.49 last 17.78) - open/close 1864 (time 1784190498.33 total 121.09 last 12.55) - open/close 1900 (time 1784190508.35 total 131.11 last 3.59) - open/close 2089 (time 1784190518.36 total 141.12 last 18.88) - open/close 2274 (time 1784190528.58 total 151.34 last 18.10) - open/close 2493 (time 1784190539.10 total 161.86 last 20.82) - open/close 2616 (time 1784190549.48 total 172.23 last 11.85) - open/close 2768 (time 1784190559.48 total 182.23 last 15.20) - open/close 2875 (time 1784190570.13 total 192.88 last 10.05) - open/close 3052 (time 1784190580.14 total 202.90 last 17.67) - open/close 3327 (time 1784190590.15 total 212.91 last 27.48) - open/close 3586 (time 1784190600.15 total 222.91 last 25.90) - open/close 3855 (time 1784190610.15 total 232.91 last 26.89) - open/close 4018 (time 1784190620.64 total 243.39 last 15.55) - open/close 4284 (time 1784190630.64 total 253.39 last 26.60) - open/close 4532 (time 1784190641.41 total 264.17 last 23.02) - open/close 4776 (time 1784190652.13 total 274.89 last 22.76) - open/close 5021 (time 1784190662.14 total 284.89 last 24.49) - open/close 5172 (time 1784190672.14 total 294.89 last 15.10) - open/close 5263 (time 1784190682.15 total 304.91 last 9.09) - open/close 5495 (time 1784190693.13 total 315.89 last 21.13) - open/close 5667 (time 1784190703.15 total 325.91 last 17.17) - open/close 5872 (time 1784190714.13 total 336.89 last 18.67) - open/close 6110 (time 1784190725.14 total 347.90 last 21.62) - open/close 6240 (time 1784190735.95 total 358.71 last 12.02) - open/close 6430 (time 1784190745.95 total 368.71 last 19.00) - open/close 6601 (time 1784190755.98 total 378.74 last 17.06) - open/close 6805 (time 1784190765.98 total 388.74 last 20.39) - open/close 7039 (time 1784190776.65 total 399.41 last 21.93) - open/close 7169 (time 1784190786.68 total 409.44 last 12.97) - open/close 7405 (time 1784190796.74 total 419.50 last 23.45) - open/close 7458 (time 1784190808.42 total 431.18 last 4.54) - open/close 7584 (time 1784190818.57 total 441.33 last 12.41) - open/close 7788 (time 1784190829.58 total 452.33 last 18.53) - open/close 7963 (time 1784190839.59 total 462.35 last 17.48) - open/close 8221 (time 1784190849.63 total 472.39 last 25.69) - open/close 8519 (time 1784190859.65 total 482.41 last 29.74) - open/close 8694 (time 1784190869.66 total 492.42 last 17.49) - open/close 8893 (time 1784190879.67 total 502.43 last 19.87) - open/close 8984 (time 1784190890.08 total 512.83 last 8.74) - open/close 9099 (time 1784190900.99 total 523.75 last 10.53) - open/close 9271 (time 1784190911.83 total 534.59 last 15.87) - open/close 9417 (time 1784190922.12 total 544.87 last 14.19) - open/close 9660 (time 1784190933.13 total 555.88 last 22.07) - open/close 9885 (time 1784190943.13 total 565.89 last 22.49) - open/close 10000 (time 1784190950.18 total 572.94 last 16.31) - open/close 10124 (time 1784190960.24 total 583.00 last 12.33) - open/close 10197 (time 1784190971.22 total 593.98 last 6.65) - open/close 10333 (time 1784190981.52 total 604.27 last 13.21) - open/close 10438 (time 1784190992.14 total 614.90 last 9.88) - open/close 10589 (time 1784191002.70 total 625.46 last 14.30) - open/close 10709 (time 1784191015.50 total 638.25 last 9.38) - open/close 10886 (time 1784191025.50 total 648.26 last 17.69) - open/close 11138 (time 1784191036.25 total 659.01 last 23.44) - open/close 11281 (time 1784191048.63 total 671.39 last 11.55) - open/close 11502 (time 1784191058.85 total 681.61 last 21.63) - open/close 11732 (time 1784191069.20 total 691.96 last 22.22) - open/close 11975 (time 1784191079.20 total 701.96 last 24.30) - open/close 12140 (time 1784191089.88 total 712.64 last 15.44) - open/close 12326 (time 1784191099.89 total 722.65 last 18.59) - open/close 12524 (time 1784191109.90 total 732.65 last 19.79) - open/close 12805 (time 1784191120.40 total 743.16 last 26.76) - open/close 13010 (time 1784191130.70 total 753.45 last 19.91) - open/close 13205 (time 1784191141.78 total 764.54 last 17.59) - open/close 13408 (time 1784191152.09 total 774.85 last 19.69) - open/close 13646 (time 1784191162.11 total 784.87 last 23.76) - open/close 13869 (time 1784191172.30 total 795.05 last 21.89) - open/close 14117 (time 1784191182.88 total 805.64 last 23.43) - open/close 14292 (time 1784191192.89 total 815.65 last 17.47) - open/close 14465 (time 1784191204.65 total 827.40 last 14.72) - open/close 14555 (time 1784191215.27 total 838.03 last 8.47) - open/close 14707 (time 1784191225.28 total 848.04 last 15.19) - open/close 14834 (time 1784191237.13 total 859.89 last 10.72) - open/close 15045 (time 1784191247.17 total 869.93 last 21.01) - open/close 15246 (time 1784191257.19 total 879.95 last 20.06) - open/close 15499 (time 1784191267.22 total 889.97 last 25.23) - open/close 15654 (time 1784191277.91 total 900.67 last 14.49) - open/close 15739 (time 1784191288.57 total 911.33 last 7.97) - open/close 15987 (time 1784191298.59 total 921.35 last 24.76) - open/close 16119 (time 1784191308.61 total 931.37 last 13.17) - open/close 16263 (time 1784191319.26 total 942.02 last 13.52) - open/close 16439 (time 1784191329.80 total 952.56 last 16.71) - open/close 16619 (time 1784191341.72 total 964.48 last 15.10) - open/close 16688 (time 1784191351.72 total 974.48 last 6.90) - open/close 16822 (time 1784191362.14 total 984.89 last 12.87) - open/close 16941 (time 1784191372.37 total 995.13 last 11.62) - open/close 17015 (time 1784191382.61 total 1005.37 last 7.23) - open/close 17186 (time 1784191392.67 total 1015.43 last 17.00) - open/close 17268 (time 1784191402.89 total 1025.65 last 8.02) - open/close 17443 (time 1784191413.96 total 1036.71 last 15.82) - open/close 17785 (time 1784191423.98 total 1046.73 last 34.13) - open/close 18432 (time 1784191433.98 total 1056.74 last 64.65) - open/close 19166 (time 1784191444.00 total 1066.76 last 73.28) - open/close 19898 (time 1784191454.02 total 1076.78 last 73.04) - open/close 20528 (time 1784191464.03 total 1086.79 last 62.93) - open/close 21265 (time 1784191474.06 total 1096.81 last 73.53) - open/close 21892 (time 1784191484.06 total 1106.82 last 62.68) - open/close 22626 (time 1784191494.06 total 1116.82 last 73.38) - open/close 23293 (time 1784191504.07 total 1126.83 last 66.65) - open/close 23968 (time 1784191514.07 total 1136.83 last 67.50) - open/close 24690 (time 1784191524.08 total 1146.84 last 72.10) - open/close 25376 (time 1784191534.09 total 1156.85 last 68.55) - open/close 26009 (time 1784191544.09 total 1166.85 last 63.27) - open/close 26680 (time 1784191554.10 total 1176.86 last 67.07) - open/close 27390 (time 1784191564.10 total 1186.86 last 70.98) - open/close 28047 (time 1784191574.10 total 1196.86 last 65.69) - open/close 28670 (time 1784191584.11 total 1206.87 last 62.24) - open/close 29304 (time 1784191594.12 total 1216.88 last 63.35) - open/close 29897 (time 1784191604.13 total 1226.89 last 59.23) - open/close 30561 (time 1784191614.14 total 1236.90 last 66.37) - open/close 31276 (time 1784191624.16 total 1246.92 last 71.35) - open/close 31964 (time 1784191634.17 total 1256.93 last 68.72) - open/close 32609 (time 1784191644.18 total 1266.93 last 64.46) - open/close 33270 (time 1784191654.21 total 1276.96 last 65.91) - open/close 33901 (time 1784191664.22 total 1286.97 last 63.03) - open/close 34594 (time 1784191674.22 total 1296.98 last 69.25) - open/close 35226 (time 1784191684.23 total 1306.99 last 63.14) - open/close 35819 (time 1784191694.24 total 1316.99 last 59.28) - open/close 36415 (time 1784191704.24 total 1327.00 last 59.55) - open/close 37138 (time 1784191714.25 total 1337.01 last 72.27) - open/close 37867 (time 1784191724.27 total 1347.02 last 72.77) - open/close 38479 (time 1784191734.30 total 1357.06 last 60.99) - open/close 39152 (time 1784191744.32 total 1367.07 last 67.19) - open/close 39863 (time 1784191754.32 total 1377.08 last 71.08) - open/close 40531 (time 1784191764.32 total 1387.08 last 66.77) - open/close 41154 (time 1784191774.33 total 1397.09 last 62.24) - open/close 41839 (time 1784191784.34 total 1407.10 last 68.42) - open/close 42472 (time 1784191794.35 total 1417.11 last 63.27) - open/close 43146 (time 1784191804.37 total 1427.13 last 67.25) - open/close 43831 (time 1784191814.37 total 1437.13 last 68.49) - open/close 44558 (time 1784191824.38 total 1447.14 last 72.65) - open/close 45160 (time 1784191834.39 total 1457.14 last 60.15) - open/close 45777 (time 1784191844.39 total 1467.15 last 61.67) - open/close 46265 (time 1784191854.41 total 1477.16 last 48.73) - open/close 46864 (time 1784191864.44 total 1487.19 last 59.72) - open/close 47509 (time 1784191874.46 total 1497.22 last 64.36) - open/close 48120 (time 1784191884.46 total 1507.22 last 61.09) - open/close 48666 (time 1784191894.46 total 1517.22 last 54.58) - open/close 49285 (time 1784191904.47 total 1527.23 last 61.85) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1531.98 seconds: 32.40 ops/second - unlinked 0 (time 1784191914 ; total 0 ; last 0) - unlinked 10000 (time 1784192084 ; total 170 ; last 170) - unlinked 20000 (time 1784192252 ; total 338 ; last 168) - unlinked 30000 (time 1784192410 ; total 496 ; last 158) - unlinked 40000 (time 1784192570 ; total 656 ; last 160) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 804 seconds: 61.730099 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg460-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg460-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg460-server stop mds service on oleg460-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg460-server stop mds service on oleg460-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg460-server unloading modules via unload_modules_local on: 'oleg460-server' oleg460-server: oleg460-server.virtnet: executing unload_modules_local oleg460-server: modules unloaded. pdsh@oleg460-client: oleg460-client: ssh exited with exit code 2 pdsh@oleg460-client: oleg460-server: ssh exited with exit code 2 pdsh@oleg460-client: oleg460-client: ssh exited with exit code 2 pdsh@oleg460-client: oleg460-server: ssh exited with exit code 2