== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:29:29 (1781224169) start mds service on oleg257-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing load_modules_local oleg257-server: Loading modules from /home/green/git/lustre-release/lustre oleg257-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg257-server: detected 4 online CPUs by sysfs oleg257-server: MODOPTS_LIBCFS= oleg257-server: Force libcfs to create 2 CPU partitions oleg257-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg257-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg257-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg257-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg257-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-OST0000 oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 132 (time 1781224302.35 total 10.27 last 12.86) - open/close 266 (time 1781224313.23 total 21.14 last 12.32) - open/close 381 (time 1781224323.29 total 31.20 last 11.43) - open/close 508 (time 1781224333.48 total 41.39 last 12.46) - open/close 602 (time 1781224344.16 total 52.07 last 8.81) - open/close 719 (time 1781224354.18 total 62.09 last 11.68) - open/close 824 (time 1781224364.19 total 72.10 last 10.49) - open/close 986 (time 1781224377.05 total 84.96 last 12.60) - open/close 1244 (time 1781224387.51 total 95.42 last 24.66) - open/close 1474 (time 1781224398.05 total 105.96 last 21.82) - open/close 1642 (time 1781224408.49 total 116.40 last 16.09) - open/close 1843 (time 1781224418.76 total 126.67 last 19.57) - open/close 1947 (time 1781224429.65 total 137.56 last 9.55) - open/close 2046 (time 1781224441.09 total 149.00 last 8.66) - open/close 2252 (time 1781224451.57 total 159.48 last 19.65) - open/close 2387 (time 1781224461.58 total 169.49 last 13.49) - open/close 2497 (time 1781224472.34 total 180.26 last 10.22) - open/close 2637 (time 1781224483.30 total 191.22 last 12.77) - open/close 2789 (time 1781224493.46 total 201.37 last 14.96) - open/close 2955 (time 1781224504.56 total 212.47 last 14.95) - open/close 3093 (time 1781224517.20 total 225.11 last 10.92) - open/close 3225 (time 1781224527.78 total 235.69 last 12.48) - open/close 3311 (time 1781224543.38 total 251.29 last 5.51) - open/close 3451 (time 1781224554.39 total 262.30 last 12.71) - open/close 3593 (time 1781224565.15 total 273.06 last 13.20) - open/close 3673 (time 1781224576.43 total 284.34 last 7.09) - open/close 3835 (time 1781224589.08 total 296.99 last 12.80) - open/close 4013 (time 1781224599.11 total 307.02 last 17.74) - open/close 4110 (time 1781224609.12 total 317.03 last 9.69) - open/close 4224 (time 1781224619.57 total 327.48 last 10.91) - open/close 4305 (time 1781224630.36 total 338.28 last 7.51) - open/close 4403 (time 1781224640.75 total 348.66 last 9.44) - open/close 4515 (time 1781224650.76 total 358.67 last 11.19) - open/close 4624 (time 1781224661.66 total 369.58 last 10.00) - open/close 4710 (time 1781224671.95 total 379.86 last 8.36) - open/close 4876 (time 1781224682.22 total 390.13 last 16.16) - open/close 4969 (time 1781224692.24 total 400.15 last 9.28) - open/close 5088 (time 1781224702.58 total 410.49 last 11.51) - open/close 5204 (time 1781224713.06 total 420.98 last 11.06) - open/close 5289 (time 1781224723.36 total 431.27 last 8.26) - open/close 5352 (time 1781224733.66 total 441.58 last 6.11) - open/close 5410 (time 1781224747.54 total 455.45 last 4.18) - open/close 5544 (time 1781224758.05 total 465.96 last 12.75) - open/close 5704 (time 1781224768.09 total 476.00 last 15.93) - open/close 5855 (time 1781224778.15 total 486.06 last 15.01) - open/close 5981 (time 1781224788.90 total 496.81 last 11.72) - open/close 6060 (time 1781224799.09 total 507.00 last 7.75) - open/close 6130 (time 1781224811.29 total 519.21 last 5.74) - open/close 6231 (time 1781224822.24 total 530.15 last 9.23) - open/close 6408 (time 1781224832.24 total 540.15 last 17.69) - open/close 6493 (time 1781224842.98 total 550.89 last 7.91) - open/close 6572 (time 1781224854.94 total 562.85 last 6.61) - open/close 6679 (time 1781224865.24 total 573.16 last 10.39) - open/close 6742 (time 1781224875.26 total 583.17 last 6.29) - open/close 6905 (time 1781224885.26 total 593.17 last 16.30) - open/close 7048 (time 1781224895.71 total 603.62 last 13.69) - open/close 7175 (time 1781224906.22 total 614.13 last 12.08) - open/close 7239 (time 1781224922.39 total 630.30 last 3.96) - open/close 7336 (time 1781224932.39 total 640.30 last 9.70) - open/close 7489 (time 1781224944.02 total 651.93 last 13.15) - open/close 7670 (time 1781224954.05 total 661.96 last 18.05) - open/close 7798 (time 1781224964.05 total 671.96 last 12.79) - open/close 7864 (time 1781224974.07 total 681.98 last 6.59) - open/close 7982 (time 1781224984.48 total 692.39 last 11.34) - open/close 8031 (time 1781224994.91 total 702.82 last 4.70) - open/close 8154 (time 1781225005.40 total 713.31 last 11.73) - open/close 8257 (time 1781225016.21 total 724.12 last 9.52) - open/close 8331 (time 1781225026.66 total 734.57 last 7.08) - open/close 8497 (time 1781225036.67 total 744.58 last 16.59) - open/close 8692 (time 1781225047.66 total 755.57 last 17.73) - open/close 8805 (time 1781225057.94 total 765.85 last 11.00) - open/close 8896 (time 1781225068.79 total 776.70 last 8.39) - open/close 9001 (time 1781225078.91 total 786.82 last 10.38) - open/close 9122 (time 1781225088.91 total 796.82 last 12.09) - open/close 9277 (time 1781225099.37 total 807.28 last 14.82) - open/close 9363 (time 1781225109.78 total 817.69 last 8.26) - open/close 9475 (time 1781225119.79 total 827.70 last 11.19) - open/close 9587 (time 1781225129.79 total 837.70 last 11.20) - open/close 9698 (time 1781225139.82 total 847.73 last 11.07) - open/close 9833 (time 1781225150.70 total 858.61 last 12.41) - open/close 10000 (time 1781225157.66 total 865.57 last 24.00) - open/close 10071 (time 1781225167.71 total 875.62 last 7.06) - open/close 10140 (time 1781225177.86 total 885.77 last 6.80) - open/close 10264 (time 1781225188.60 total 896.51 last 11.54) - open/close 10361 (time 1781225198.61 total 906.52 last 9.69) - open/close 10483 (time 1781225209.14 total 917.05 last 11.58) - open/close 10534 (time 1781225219.94 total 927.85 last 4.72) - open/close 10708 (time 1781225231.22 total 939.13 last 15.43) - open/close 10798 (time 1781225241.89 total 949.80 last 8.43) - open/close 10885 (time 1781225254.39 total 962.30 last 6.96) - open/close 11005 (time 1781225265.74 total 973.65 last 10.57) - open/close 11162 (time 1781225277.58 total 985.50 last 13.26) - open/close 11234 (time 1781225287.59 total 995.50 last 7.20) - open/close 11342 (time 1781225297.63 total 1005.54 last 10.76) - open/close 11436 (time 1781225308.32 total 1016.23 last 8.80) - open/close 11594 (time 1781225318.32 total 1026.23 last 15.79) - open/close 11740 (time 1781225328.85 total 1036.77 last 13.86) - open/close 11814 (time 1781225340.79 total 1048.70 last 6.20) - open/close 11924 (time 1781225350.81 total 1058.72 last 10.98) - open/close 12043 (time 1781225361.68 total 1069.59 last 10.94) - open/close 12262 (time 1781225371.69 total 1079.60 last 21.87) - open/close 12394 (time 1781225381.74 total 1089.65 last 13.15) - open/close 12609 (time 1781225391.76 total 1099.67 last 21.45) - open/close 12668 (time 1781225401.78 total 1109.69 last 5.89) - open/close 12728 (time 1781225412.82 total 1120.73 last 5.43) - open/close 12832 (time 1781225423.26 total 1131.17 last 9.96) - open/close 12931 (time 1781225433.27 total 1141.18 last 9.89) - open/close 12989 (time 1781225444.27 total 1152.18 last 5.27) - open/close 13086 (time 1781225455.08 total 1163.00 last 8.97) - open/close 13188 (time 1781225465.09 total 1173.00 last 10.19) - open/close 13308 (time 1781225475.29 total 1183.20 last 11.77) - open/close 13368 (time 1781225486.44 total 1194.35 last 5.38) - open/close 13871 (time 1781225496.44 total 1204.35 last 50.29) - open/close 14472 (time 1781225506.44 total 1214.35 last 60.10) - open/close 15088 (time 1781225516.46 total 1224.37 last 61.49) - open/close 15742 (time 1781225526.46 total 1234.37 last 65.39) - open/close 16375 (time 1781225536.48 total 1244.39 last 63.23) - open/close 16976 (time 1781225546.50 total 1254.41 last 59.98) - open/close 17654 (time 1781225556.50 total 1264.41 last 67.80) - open/close 18225 (time 1781225566.50 total 1274.41 last 57.10) - open/close 18784 (time 1781225576.51 total 1284.42 last 55.81) - open/close 19392 (time 1781225586.51 total 1294.42 last 60.80) - open/close 20000 (time 1781225595.89 total 1303.80 last 64.82) - open/close 20672 (time 1781225605.90 total 1313.81 last 67.15) - open/close 21332 (time 1781225615.91 total 1323.82 last 65.94) - open/close 21941 (time 1781225625.92 total 1333.83 last 60.82) - open/close 22529 (time 1781225635.94 total 1343.85 last 58.71) - open/close 23182 (time 1781225645.95 total 1353.87 last 65.19) - open/close 23862 (time 1781225655.96 total 1363.87 last 67.94) - open/close 24605 (time 1781225665.96 total 1373.87 last 74.30) - open/close 25297 (time 1781225675.96 total 1383.88 last 69.20) - open/close 26123 (time 1781225685.97 total 1393.88 last 82.59) - open/close 26889 (time 1781225695.97 total 1403.88 last 76.59) - open/close 27542 (time 1781225705.99 total 1413.90 last 65.16) - open/close 28209 (time 1781225715.99 total 1423.90 last 66.67) - open/close 28975 (time 1781225726.00 total 1433.91 last 76.56) - open/close 29567 (time 1781225736.00 total 1443.91 last 59.18) - open/close 30000 (time 1781225742.28 total 1450.19 last 68.95) - open/close 30650 (time 1781225752.29 total 1460.20 last 64.92) - open/close 31167 (time 1781225762.30 total 1470.21 last 51.65) - open/close 31776 (time 1781225772.31 total 1480.22 last 60.86) - open/close 32550 (time 1781225782.31 total 1490.22 last 77.39) - open/close 33258 (time 1781225792.32 total 1500.23 last 70.74) - open/close 33953 (time 1781225802.33 total 1510.24 last 69.45) - open/close 34598 (time 1781225812.34 total 1520.25 last 64.40) - open/close 35060 (time 1781225822.35 total 1530.26 last 46.16) - open/close 35472 (time 1781225832.37 total 1540.28 last 41.14) - open/close 36120 (time 1781225842.37 total 1550.28 last 64.78) - open/close 36686 (time 1781225852.37 total 1560.28 last 56.59) - open/close 37287 (time 1781225862.38 total 1570.29 last 60.08) - open/close 37884 (time 1781225872.38 total 1580.29 last 59.68) - open/close 38407 (time 1781225882.38 total 1590.29 last 52.28) - open/close 39199 (time 1781225892.39 total 1600.31 last 79.10) - open/close 39865 (time 1781225902.40 total 1610.31 last 66.59) - open/close 40000 (time 1781225904.41 total 1612.32 last 67.11) - open/close 40679 (time 1781225914.41 total 1622.32 last 67.87) - open/close 41391 (time 1781225924.41 total 1632.33 last 71.18) - open/close 42020 (time 1781225934.42 total 1642.33 last 62.87) - open/close 42584 (time 1781225944.46 total 1652.37 last 56.18) - open/close 43308 (time 1781225954.48 total 1662.39 last 72.28) - open/close 43989 (time 1781225964.48 total 1672.39 last 68.10) - open/close 44570 (time 1781225974.48 total 1682.39 last 58.08) - open/close 45173 (time 1781225984.50 total 1692.41 last 60.17) - open/close 45648 (time 1781225994.50 total 1702.42 last 47.48) - open/close 46118 (time 1781226004.51 total 1712.42 last 46.96) - open/close 46657 (time 1781226014.53 total 1722.44 last 53.81) - open/close 47281 (time 1781226024.54 total 1732.45 last 62.36) - open/close 47978 (time 1781226034.54 total 1742.45 last 69.67) - open/close 48606 (time 1781226044.56 total 1752.47 last 62.68) - open/close 49267 (time 1781226054.58 total 1762.49 last 65.98) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1769.03 seconds: 28.06 ops/second - unlinked 0 (time 1781226067 ; total 0 ; last 0) - unlinked 10000 (time 1781226227 ; total 160 ; last 160) - unlinked 20000 (time 1781226374 ; total 307 ; last 147) - unlinked 30000 (time 1781226534 ; total 467 ; last 160) - unlinked 40000 (time 1781226680 ; total 613 ; last 146) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 759 seconds: 65.389984 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg257-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg257-server unloading modules via unload_modules_local on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing unload_modules_local oleg257-server: modules unloaded. pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-client: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-client: ssh exited with exit code 2