== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:52:31 (1709787151) start mds service on oleg341-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg341-server: oleg341-server.virtnet: executing set_default_debug -1 all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg341-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg341-server: oleg341-server.virtnet: executing set_default_debug -1 all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Started lustre-OST0000 oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg341-client.virtnet: -o user_xattr,flock oleg341-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 183 (time 1709787185.43 total 10.36 last 17.66) - open/close 345 (time 1709787196.21 total 21.14 last 15.03) - open/close 568 (time 1709787206.89 total 31.82 last 20.88) - open/close 726 (time 1709787217.34 total 42.27 last 15.12) - open/close 979 (time 1709787227.80 total 52.73 last 24.19) - open/close 1202 (time 1709787238.15 total 63.09 last 21.53) - open/close 1408 (time 1709787248.16 total 73.09 last 20.59) - open/close 1521 (time 1709787260.15 total 85.09 last 9.42) - open/close 1824 (time 1709787270.84 total 95.78 last 28.35) - open/close 2015 (time 1709787280.93 total 105.86 last 18.94) - open/close 2221 (time 1709787290.93 total 115.86 last 20.60) - open/close 2357 (time 1709787301.82 total 126.76 last 12.48) - open/close 2487 (time 1709787312.09 total 137.02 last 12.66) - open/close 2747 (time 1709787322.94 total 147.88 last 23.96) - open/close 2963 (time 1709787333.67 total 158.60 last 20.13) - open/close 3130 (time 1709787343.92 total 168.86 last 16.29) - open/close 3472 (time 1709787354.62 total 179.56 last 31.96) - open/close 3650 (time 1709787364.73 total 189.67 last 17.61) - open/close 3908 (time 1709787374.73 total 199.67 last 25.79) - open/close 4109 (time 1709787385.53 total 210.46 last 18.62) - open/close 4363 (time 1709787395.56 total 220.49 last 25.33) - open/close 4504 (time 1709787405.58 total 230.51 last 14.07) - open/close 4786 (time 1709787415.60 total 240.53 last 28.15) - open/close 4954 (time 1709787426.38 total 251.31 last 15.58) - open/close 5114 (time 1709787437.39 total 262.32 last 14.53) - open/close 5264 (time 1709787447.92 total 272.85 last 14.25) - open/close 5369 (time 1709787457.94 total 282.87 last 10.48) - open/close 5575 (time 1709787467.98 total 292.92 last 20.50) - open/close 5880 (time 1709787477.99 total 302.92 last 30.48) - open/close 6073 (time 1709787488.43 total 313.37 last 18.48) - open/close 6300 (time 1709787498.44 total 323.37 last 22.68) - open/close 6449 (time 1709787508.46 total 333.40 last 14.87) - open/close 6612 (time 1709787519.19 total 344.13 last 15.19) - open/close 6790 (time 1709787529.21 total 354.15 last 17.77) - open/close 6942 (time 1709787539.85 total 364.79 last 14.28) - open/close 7075 (time 1709787550.61 total 375.54 last 12.37) - open/close 7236 (time 1709787560.64 total 385.58 last 16.04) - open/close 7363 (time 1709787571.03 total 395.97 last 12.23) - open/close 7527 (time 1709787581.04 total 405.98 last 16.39) - open/close 7661 (time 1709787591.93 total 416.87 last 12.30) - open/close 7842 (time 1709787601.96 total 426.89 last 18.06) - open/close 8042 (time 1709787611.96 total 436.90 last 20.00) - open/close 8231 (time 1709787621.96 total 446.90 last 18.90) - open/close 8367 (time 1709787631.97 total 456.90 last 13.60) - open/close 8613 (time 1709787641.97 total 466.90 last 24.60) - open/close 8822 (time 1709787652.72 total 477.65 last 19.44) - open/close 8950 (time 1709787663.24 total 488.18 last 12.16) - open/close 9065 (time 1709787673.84 total 498.77 last 10.85) - open/close 9232 (time 1709787684.03 total 508.96 last 16.39) - open/close 9411 (time 1709787694.03 total 518.97 last 17.89) - open/close 9552 (time 1709787704.07 total 529.00 last 14.05) - open/close 9781 (time 1709787714.11 total 539.04 last 22.81) - open/close 9905 (time 1709787724.47 total 549.40 last 11.97) - open/close 10000 (time 1709787730.46 total 555.39 last 15.85) - open/close 10130 (time 1709787741.24 total 566.18 last 12.06) - open/close 10294 (time 1709787751.28 total 576.21 last 16.35) - open/close 10483 (time 1709787761.74 total 586.67 last 18.07) - open/close 10695 (time 1709787772.34 total 597.28 last 19.99) - open/close 10839 (time 1709787782.52 total 607.46 last 14.15) - open/close 11051 (time 1709787792.52 total 617.46 last 21.20) - open/close 11298 (time 1709787802.53 total 627.47 last 24.68) - open/close 11884 (time 1709787812.55 total 637.48 last 58.50) - open/close 12398 (time 1709787822.56 total 647.50 last 51.33) - open/close 12963 (time 1709787832.57 total 657.50 last 56.46) - open/close 13491 (time 1709787842.59 total 667.52 last 52.70) - open/close 14029 (time 1709787852.61 total 677.55 last 53.65) - open/close 14546 (time 1709787862.63 total 687.56 last 51.64) - open/close 15049 (time 1709787872.64 total 697.57 last 50.24) - open/close 15522 (time 1709787882.65 total 707.58 last 47.25) - open/close 15975 (time 1709787892.66 total 717.59 last 45.26) - open/close 16435 (time 1709787902.69 total 727.62 last 45.87) - open/close 16900 (time 1709787912.70 total 737.63 last 46.45) - open/close 17324 (time 1709787922.70 total 747.63 last 42.39) - open/close 17734 (time 1709787932.72 total 757.65 last 40.92) - open/close 18139 (time 1709787942.72 total 767.66 last 40.49) - open/close 18544 (time 1709787952.73 total 777.66 last 40.48) - open/close 18953 (time 1709787962.75 total 787.68 last 40.81) - open/close 19356 (time 1709787972.75 total 797.68 last 40.29) - open/close 19775 (time 1709787982.76 total 807.69 last 41.85) - open/close 20000 (time 1709787987.68 total 812.61 last 45.73) - open/close 20446 (time 1709787997.68 total 822.62 last 44.58) - open/close 20861 (time 1709788007.70 total 832.64 last 41.43) - open/close 21246 (time 1709788017.71 total 842.64 last 38.47) - open/close 21632 (time 1709788027.72 total 852.66 last 38.55) - open/close 21994 (time 1709788037.74 total 862.68 last 36.12) - open/close 22339 (time 1709788047.76 total 872.70 last 34.44) - open/close 22675 (time 1709788057.77 total 882.71 last 33.57) - open/close 23026 (time 1709788067.77 total 892.71 last 35.10) - open/close 23430 (time 1709788077.79 total 902.72 last 40.33) - open/close 23908 (time 1709788087.81 total 912.75 last 47.69) - open/close 24360 (time 1709788097.83 total 922.77 last 45.10) - open/close 24828 (time 1709788107.84 total 932.77 last 46.77) - open/close 25265 (time 1709788117.86 total 942.79 last 43.63) - open/close 25717 (time 1709788127.87 total 952.80 last 45.15) - open/close 26173 (time 1709788137.87 total 962.80 last 45.60) - open/close 26615 (time 1709788147.88 total 972.81 last 44.15) - open/close 27046 (time 1709788157.90 total 982.83 last 43.02) - open/close 27469 (time 1709788167.90 total 992.84 last 42.28) - open/close 27891 (time 1709788177.92 total 1002.86 last 42.11) - open/close 28308 (time 1709788187.94 total 1012.87 last 41.64) - open/close 28713 (time 1709788197.95 total 1022.89 last 40.42) - open/close 29088 (time 1709788207.96 total 1032.90 last 37.47) - open/close 29485 (time 1709788217.97 total 1042.90 last 39.67) - open/close 29901 (time 1709788227.97 total 1052.90 last 41.60) - open/close 30000 (time 1709788230.46 total 1055.39 last 39.82) - open/close 30383 (time 1709788240.47 total 1065.41 last 38.24) - open/close 30753 (time 1709788250.49 total 1075.43 last 36.93) - open/close 31100 (time 1709788260.51 total 1085.45 last 34.63) - open/close 31455 (time 1709788270.51 total 1095.45 last 35.50) - open/close 31807 (time 1709788280.52 total 1105.45 last 35.19) - open/close 32152 (time 1709788290.56 total 1115.49 last 34.36) - open/close 32501 (time 1709788300.59 total 1125.52 last 34.79) - open/close 32843 (time 1709788310.59 total 1135.53 last 34.19) - open/close 33185 (time 1709788320.59 total 1145.53 last 34.20) - open/close 33553 (time 1709788330.63 total 1155.57 last 36.66) - open/close 33906 (time 1709788340.64 total 1165.57 last 35.28) - open/close 34261 (time 1709788350.65 total 1175.59 last 35.44) - open/close 34608 (time 1709788360.67 total 1185.60 last 34.65) - open/close 34911 (time 1709788370.68 total 1195.61 last 30.28) - open/close 35232 (time 1709788380.70 total 1205.64 last 32.02) - open/close 35567 (time 1709788390.72 total 1215.65 last 33.45) - open/close 35904 (time 1709788400.73 total 1225.67 last 33.65) - open/close 36231 (time 1709788410.75 total 1235.69 last 32.64) - open/close 36258 (time 1709788422.66 total 1247.59 last 2.27) - open/close 36564 (time 1709788432.67 total 1257.60 last 30.57) - open/close 36881 (time 1709788442.68 total 1267.61 last 31.66) - open/close 37184 (time 1709788452.69 total 1277.62 last 30.27) - open/close 37504 (time 1709788462.72 total 1287.65 last 31.90) - open/close 37814 (time 1709788472.74 total 1297.68 last 30.93) - open/close 38121 (time 1709788482.76 total 1307.69 last 30.65) - open/close 38397 (time 1709788492.77 total 1317.70 last 27.58) - open/close 38686 (time 1709788502.77 total 1327.70 last 28.90) - open/close 38984 (time 1709788512.77 total 1337.71 last 29.78) - open/close 39280 (time 1709788522.79 total 1347.73 last 29.54) - open/close 39574 (time 1709788532.80 total 1357.74 last 29.37) - open/close 39869 (time 1709788542.80 total 1367.74 last 29.50) - open/close 40000 (time 1709788547.16 total 1372.09 last 30.08) - open/close 40286 (time 1709788557.18 total 1382.12 last 28.54) - open/close 40578 (time 1709788567.21 total 1392.14 last 29.11) - open/close 40880 (time 1709788577.23 total 1402.16 last 30.15) - open/close 41179 (time 1709788587.26 total 1412.19 last 29.81) - open/close 41472 (time 1709788597.28 total 1422.21 last 29.24) - open/close 41770 (time 1709788607.30 total 1432.23 last 29.74) - open/close 42060 (time 1709788617.30 total 1442.23 last 29.00) - open/close 42361 (time 1709788627.31 total 1452.24 last 30.07) - open/close 42648 (time 1709788637.33 total 1462.26 last 28.63) - open/close 42943 (time 1709788647.36 total 1472.29 last 29.43) - open/close 43244 (time 1709788657.37 total 1482.30 last 30.07) - open/close 43535 (time 1709788667.38 total 1492.31 last 29.07) - open/close 43818 (time 1709788677.39 total 1502.32 last 28.27) - open/close 44083 (time 1709788687.39 total 1512.33 last 26.49) - open/close 44359 (time 1709788697.42 total 1522.35 last 27.52) - open/close 44639 (time 1709788707.43 total 1532.37 last 27.96) - open/close 44910 (time 1709788717.45 total 1542.38 last 27.06) - open/close 45193 (time 1709788727.47 total 1552.40 last 28.25) - open/close 45475 (time 1709788737.48 total 1562.42 last 28.15) - open/close 45759 (time 1709788747.48 total 1572.42 last 28.40) - open/close 46025 (time 1709788757.52 total 1582.45 last 26.51) - open/close 46285 (time 1709788767.52 total 1592.46 last 25.99) - open/close 46530 (time 1709788777.53 total 1602.46 last 24.49) - open/close 46787 (time 1709788787.54 total 1612.47 last 25.68) - open/close 47035 (time 1709788797.57 total 1622.51 last 24.71) - open/close 47283 (time 1709788807.58 total 1632.52 last 24.77) - open/close 47531 (time 1709788817.62 total 1642.55 last 24.72) - open/close 47768 (time 1709788827.66 total 1652.59 last 23.60) - open/close 48019 (time 1709788837.66 total 1662.59 last 25.10) - open/close 48265 (time 1709788847.69 total 1672.63 last 24.52) - open/close 48507 (time 1709788857.72 total 1682.66 last 24.13) - open/close 48746 (time 1709788867.73 total 1692.66 last 23.89) - open/close 48991 (time 1709788877.77 total 1702.71 last 24.39) - open/close 49240 (time 1709788887.80 total 1712.74 last 24.83) - open/close 49505 (time 1709788897.82 total 1722.76 last 26.44) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 1730.72 seconds: 28.72 ops/second - unlinked 0 (time 1709788907 ; total 0 ; last 0) - unlinked 10000 (time 1709789385 ; total 478 ; last 478) - unlinked 20000 (time 1709789813 ; total 906 ; last 428) - unlinked 30000 (time 1709790127 ; total 1220 ; last 314) - unlinked 40000 (time 1709790281 ; total 1374 ; last 154) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 1422 seconds: 34.949368 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg341-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg341-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg341-server stop mds service on oleg341-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg341-server unloading modules on: 'oleg341-server' oleg341-server: oleg341-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg341-client: oleg341-client: ssh exited with exit code 2 pdsh@oleg341-client: oleg341-server: ssh exited with exit code 2 pdsh@oleg341-client: oleg341-client: ssh exited with exit code 2 pdsh@oleg341-client: oleg341-server: ssh exited with exit code 2